Jul 27 16:14:51.645640 ip-10-0-140-199 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 16:14:51.645651 ip-10-0-140-199 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 16:14:51.645658 ip-10-0-140-199 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 16:14:51.645863 ip-10-0-140-199 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 16:15:01.706585 ip-10-0-140-199 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 16:15:01.706604 ip-10-0-140-199 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d330a1c3dc614cb4ad42f11688eb461c --
Jul 27 16:16:39.979582 ip-10-0-140-199 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 16:16:40.392129 ip-10-0-140-199 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:16:40.392129 ip-10-0-140-199 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 16:16:40.392129 ip-10-0-140-199 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:16:40.392129 ip-10-0-140-199 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 16:16:40.392129 ip-10-0-140-199 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:16:40.393117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.393034    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 16:16:40.399465 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399441    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:16:40.399465 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399459    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:16:40.399465 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399463    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:16:40.399465 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399467    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:16:40.399465 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399472    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:16:40.399465 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399475    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399479    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399482    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399485    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399487    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399490    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399493    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399496    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399498    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399503    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399507    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399509    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399512    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399515    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399518    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399521    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399524    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399527    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399529    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:16:40.399709 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399532    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399535    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399537    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399540    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399543    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399546    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399549    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399552    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399554    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399557    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399559    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399562    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399565    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399568    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399571    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399574    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399577    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399580    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399582    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399585    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:16:40.400161 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399587    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399590    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399593    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399595    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399598    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399600    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399603    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399605    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399609    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399611    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399617    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399620    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399624    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399626    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399629    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399632    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399635    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399637    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399640    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399643    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:16:40.400693 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399646    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399648    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399651    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399654    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399658    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399661    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399664    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399667    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399670    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399672    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399675    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399678    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399681    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399683    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399686    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399689    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399691    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399694    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399697    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399701    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:16:40.401169 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399704    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399706    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399710    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399713    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399716    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399718    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399722    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399725    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399729    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399732    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399734    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399737    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399741    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399746    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.399750    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400515    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400529    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400533    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400537    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:16:40.401782 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400540    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400542    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400545    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400549    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400551    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400554    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400556    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400559    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400562    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400565    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400567    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400570    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400574    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400579    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400582    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400585    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400587    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400590    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400593    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:16:40.402241 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400595    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400599    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400601    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400604    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400607    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400610    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400612    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400615    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400617    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400620    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400624    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400627    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400629    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400632    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400635    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400637    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400640    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400643    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400645    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400648    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400650    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:16:40.402733 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400653    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400657    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400660    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400664    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400666    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400669    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400672    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400674    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400677    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400680    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400682    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400685    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400687    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400690    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400693    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400696    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400699    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400701    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400704    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400707    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:16:40.403266 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400709    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400712    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400715    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400718    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400721    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400723    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400726    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400728    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400731    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400733    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400736    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400739    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400742    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400748    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400751    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400754    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400757    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400759    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400762    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:16:40.403768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400766    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400769    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400771    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400775    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400777    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400780    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400782    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400785    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400788    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400790    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400793    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400796    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400799    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400802    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400805    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.400808    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401173    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401178    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401182    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401185    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:16:40.404220 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401188    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401191    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401194    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401197    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401200    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401203    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401205    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401208    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401210    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401213    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401215    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401218    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401222    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401224    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401228    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401231    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401233    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401236    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401239    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401241    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:16:40.404734 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401244    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401246    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401249    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401252    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401254    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401257    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401260    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401263    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401265    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401268    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401272    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401274    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401291    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401293    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401296    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401299    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401302    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401304    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401307    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:16:40.405208 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401309    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401312    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401314    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401317    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401320    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401323    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401328    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401330    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401333    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401336    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401338    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401341    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401343    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401346    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401349    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401352    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401354    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401357    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401360    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401363    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:16:40.405678 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401365    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401368    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401371    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401374    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401379    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401382    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401385    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401388    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401391    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401393    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401396    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401399    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401402    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401404    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401407    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401410    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401412    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401415    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401417    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:16:40.406195 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401421    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401423    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401426    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401428    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401431    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401434    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401438    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401441    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401445    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401448    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401451    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401453    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401456    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401459    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401462    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401465    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.401467    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401531    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401538    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401544    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 16:16:40.406671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401549    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401553    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401556    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401560    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401566    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401569    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401572    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401575    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401579    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401582    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401585    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401588    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401595    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401599    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401602    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401607    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401610    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401613    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401616    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401620    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401624    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401627    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401631    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 16:16:40.407152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401634    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401637    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401640    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401643    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401646    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401649    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401653    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401656    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401659    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401661    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401664    2559 flags.go:64] FLAG: --enable-server="true"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401667    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401671    2559 flags.go:64] FLAG: --event-burst="100"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401674    2559 flags.go:64] FLAG: --event-qps="50"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401677    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401680    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401683    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401687    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401690    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401693    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401696    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401700    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401703    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401706    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401709    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 16:16:40.407713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401712    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401715    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401718    2559 flags.go:64] FLAG: --feature-gates=""
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401723    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401726    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401729    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401732    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401735    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401738    2559 flags.go:64] FLAG: --help="false"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401741    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401744    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401747    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401750    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401753    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401757    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401760    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401762    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401765    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401768    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401771    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401774    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401777    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401780    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401782    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 16:16:40.408376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401785    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401788    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401791    2559 flags.go:64] FLAG: --lock-file=""
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401794    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401798    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401801    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401806    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401810    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401813    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401815    2559 flags.go:64] FLAG: --logging-format="text"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401818    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401823    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401825    2559 flags.go:64] FLAG: --manifest-url=""
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401828    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401833    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401836    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401841    2559 flags.go:64] FLAG: --max-pods="110"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401844    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401847    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401850    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401853    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401856    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401858    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401861    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401868    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 16:16:40.408998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401871    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401874    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401878    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401880    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401885    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401888    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401891    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401895    2559 flags.go:64] FLAG: --port="10250"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401898    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401901    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0201794d90f9abebb"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401904    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401907    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401911    2559 flags.go:64] FLAG: --register-node="true"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401914    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401917    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401921    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401924    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401927    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401932    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401935    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401938    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401940    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401943    2559 flags.go:64] FLAG: --runonce="false"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401946    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401949    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 16:16:40.409624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401952    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401955    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401958    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401960    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401964    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401967    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401969    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401972    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401975    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401978    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401981    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401984    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401986    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401992    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401994    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.401997    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402001    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402004    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402007    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402011    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402014    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402017    2559 flags.go:64] FLAG: --v="2"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402021    2559 flags.go:64] FLAG: --version="false"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402026    2559 flags.go:64] FLAG: --vmodule=""
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402030    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 16:16:40.410226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.402034    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402156    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402159    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402162    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402165    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402168    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402171    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402174    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402177    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402180    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402183    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402185    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402188    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402190    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402193    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402196    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402198    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402201    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402204    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402207    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:16:40.410877 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402211    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402213    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402216    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402219    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402221    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402223    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402227    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402230    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402232    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402235    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402238    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402241    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402244    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402247    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402250    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402253    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402256    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402259    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402261    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402264    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:16:40.411369 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402266    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402269    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402272    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402274    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402293    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402296    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402300    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402303    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402306    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402309    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402312    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402315    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402318    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402321    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402324    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402326    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402329    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402333    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402338    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:16:40.411851 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402341    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402344    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402347    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402350    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402352    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402357    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402359    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402362    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402365    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402367    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402370    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402372    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402375    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402377    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402380    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402382    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402385    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402387    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402390    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:16:40.412353 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402393    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402395    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402397    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402400    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402402    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402405    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402407    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402410    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402413    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402415    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402418    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402421    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402425    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402427    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402430    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402433    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402435    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402438    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402442    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402449    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:16:40.412824 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402451    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.402454    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.403025    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.409628    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.409645    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.409663    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.409669    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409727    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409732    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409736    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409739    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409742    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409744    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409747    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409749    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:16:40.413314 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409752    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409755    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409757    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409760    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409762    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409766    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409770    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409773    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409776    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409780    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409782    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409785    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409788    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409791    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409794    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409796    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409799    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409802    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409805    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:16:40.413690 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409807    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409810    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409823    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409826    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409829    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409831    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409834    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409837    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409839    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409842    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409845    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409848    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409850    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409853    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409856    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409858    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409861    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409863    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409865    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409868    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:16:40.414144 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409870    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409873    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409875    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409878    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409880    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409883    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409885    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409888    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409890    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409893    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409896    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409898    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409901    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409903    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409906    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409914    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409917    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409919    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409922    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:16:40.414708 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409926    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409929    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409932    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409935    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409938    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409940    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409943    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409945    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409948    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409951    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409955    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409958    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409960    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409963    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409965    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409968    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409970    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409973    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409975    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409978    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:16:40.415170 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409980    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409982    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409985    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409988    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409991    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409993    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409995    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.409998    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410001    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410009    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410012    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410014    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410025    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.410030    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410169    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:16:40.415703 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410175    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410178    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410181    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410183    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410186    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410188    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410191    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410193    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410196    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410198    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410201    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410203    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410206    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410209    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410211    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410214    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410217    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410219    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410222    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410224    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:16:40.416057 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410227    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410230    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410233    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410235    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410238    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410241    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410244    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410252    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410255    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410257    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410259    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410262    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410264    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410267    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410270    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410274    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410289    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410292    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410294    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:16:40.416554 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410297    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410299    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410302    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410304    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410307    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410310    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410312    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410315    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410317    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410320    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410322    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410326    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410330    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410333    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410335    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410339    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410343    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410346    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410348    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:16:40.417017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410351    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410355    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410369    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410372    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410375    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410377    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410380    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410382    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410385    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410388    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410390    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410392    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410395    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410398    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410400    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410403    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410405    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410407    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410410    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410412    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:16:40.417479 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410415    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410417    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410419    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410422    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410425    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410428    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410430    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410433    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410435    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410438    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410440    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410442    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410445    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410448    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410450    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410460    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410462    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410465    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410467    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:16:40.417947 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:40.410470    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:16:40.418406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.410475    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:16:40.418406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.410598    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 16:16:40.418406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.414064    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 16:16:40.418406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.415883    2559 server.go:1019] "Starting client certificate rotation"
Jul 27 16:16:40.418406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.415998    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 16:16:40.418406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.416031    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 16:16:40.418560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.418485    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 16:16:40.425464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.425435    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 16:16:40.435303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.435270    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 27 16:16:40.438001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.437989    2559 log.go:25] "Validated CRI v1 image API"
Jul 27 16:16:40.439706 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.439687    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 16:16:40.442063 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.442043    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 beadb66d-5393-43c6-96a3-ceb1faa28a75:/dev/nvme0n1p4 e668fdbf-7665-438f-803b-8ca3bfcab759:/dev/nvme0n1p3]
Jul 27 16:16:40.442131 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.442061    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 16:16:40.446651 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.446542    2559 manager.go:216] Machine: {Timestamp:2026-07-27 16:16:40.446040423 +0000 UTC m=+0.347180862 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099700 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b946bacd6ab2c14366e5a78b40c5d SystemUUID:ec2b946b-acd6-ab2c-1436-6e5a78b40c5d BootID:d330a1c3-dc61-4cb4-ad42-f11688eb461c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:6d:80:49:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:6d:80:49:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:cd:9e:52:ff:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 16:16:40.446651 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.446645    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 16:16:40.446760 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.446722    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 16:16:40.447353 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.447337    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 16:16:40.450590 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.450557    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 16:16:40.450735 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.450595    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 16:16:40.451345 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.451335    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 16:16:40.451385 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.451348    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 16:16:40.451385 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.451355    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 16:16:40.451385 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.451366    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 16:16:40.452126 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.452117    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 16:16:40.453977 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.453965    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 16:16:40.454140 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454131    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 16:16:40.454176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454160    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 16:16:40.454176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454174    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 16:16:40.454226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454185    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 16:16:40.454226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454194    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 27 16:16:40.454226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454202    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 16:16:40.454707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454694    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 16:16:40.454766 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.454713    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 16:16:40.456987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.456970    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 16:16:40.458831 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.458818    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 16:16:40.458873 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.458849    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 16:16:40.460465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460453    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 16:16:40.460501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460469    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 16:16:40.460501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460476    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 16:16:40.460501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460481    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 16:16:40.460501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460490    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 16:16:40.460501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460495    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 16:16:40.460501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460503    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 16:16:40.460748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460508    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 16:16:40.460748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460515    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 16:16:40.460748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460529    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 16:16:40.460748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460538    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 16:16:40.460748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.460543    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 16:16:40.461489 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.461478    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 16:16:40.461542 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.461491    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 16:16:40.464851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.464834    2559 server.go:1268] "Started kubelet"
Jul 27 16:16:40.464936 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.464889    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 16:16:40.467992 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.467902    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 16:16:40.468092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.468027    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 16:16:40.468557 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.468478    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 16:16:40.468557 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.468501    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 16:16:40.468715 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.468275    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-199.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 16:16:40.468652 ip-10-0-140-199 systemd[1]: Started Kubernetes Kubelet.
Jul 27 16:16:40.471900 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.471883    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 16:16:40.472032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.472019    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 16:16:40.474123 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.474101    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 16:16:40.474692 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.473723    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf68be2b81  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.464788353 +0000 UTC m=+0.365928792,LastTimestamp:2026-07-27 16:16:40.464788353 +0000 UTC m=+0.365928792,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.477692 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.477677    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 16:16:40.477692 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.477686    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 16:16:40.477813 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.477764    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.478291 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478259    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 16:16:40.478359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478352    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 16:16:40.478444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478261    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 16:16:40.478500 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478486    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 16:16:40.478500 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478494    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 16:16:40.478582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478492    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 16:16:40.478582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478513    2559 factory.go:55] Registering systemd factory
Jul 27 16:16:40.478582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478521    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 27 16:16:40.478963 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478950    2559 factory.go:153] Registering CRI-O factory
Jul 27 16:16:40.478963 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478965    2559 factory.go:223] Registration of the crio container factory successfully
Jul 27 16:16:40.479082 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.478986    2559 factory.go:103] Registering Raw factory
Jul 27 16:16:40.479082 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.479000    2559 manager.go:1196] Started watching for new ooms in manager
Jul 27 16:16:40.479476 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.479463    2559 manager.go:349] Starting recovery of all containers
Jul 27 16:16:40.482782 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.482756    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 16:16:40.488772 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.488747    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 16:16:40.489984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.489967    2559 manager.go:354] Recovery completed
Jul 27 16:16:40.497816 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.497801    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.499607 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.499575    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.499674 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.499621    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.499674 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.499632    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.500158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.500143    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 16:16:40.500158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.500157    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 16:16:40.500244 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.500174    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 16:16:40.500244 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.500183    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.501662 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.501648    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.501720 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.501677    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.501720 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.501691    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.501836 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.501824    2559 policy_none.go:49] "None policy: Start"
Jul 27 16:16:40.501880 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.501840    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 16:16:40.501880 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.501851    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 16:16:40.501962 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.501832    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf6ad174e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-199.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.499606757 +0000 UTC m=+0.400747195,LastTimestamp:2026-07-27 16:16:40.499606757 +0000 UTC m=+0.400747195,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.510821 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.510801    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cq7d6"
Jul 27 16:16:40.513128 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.513049    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf6ad1c088  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-199.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.49962612 +0000 UTC m=+0.400766559,LastTimestamp:2026-07-27 16:16:40.49962612 +0000 UTC m=+0.400766559,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.515707 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.515647    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf6ad1e650  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-199.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.499635792 +0000 UTC m=+0.400776230,LastTimestamp:2026-07-27 16:16:40.499635792 +0000 UTC m=+0.400776230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.521520 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.521503    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cq7d6"
Jul 27 16:16:40.522811 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.522732    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-199.ec2.internal.18c631bf6ad174e5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf6ad174e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-199.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.499606757 +0000 UTC m=+0.400747195,LastTimestamp:2026-07-27 16:16:40.501661995 +0000 UTC m=+0.402802440,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.525870 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.525814    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-199.ec2.internal.18c631bf6ad1c088\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf6ad1c088  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-199.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.49962612 +0000 UTC m=+0.400766559,LastTimestamp:2026-07-27 16:16:40.501684947 +0000 UTC m=+0.402825386,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.533653 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.533590    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-199.ec2.internal.18c631bf6ad1e650\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf6ad1e650  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-199.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.499635792 +0000 UTC m=+0.400776230,LastTimestamp:2026-07-27 16:16:40.501694672 +0000 UTC m=+0.402835112,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.648730 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.578830    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.648730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.647865    2559 policy_none.go:47] "Start"
Jul 27 16:16:40.678994 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.678974    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.681847 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.681836    2559 manager.go:337] "Starting Device Plugin manager"
Jul 27 16:16:40.681897 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.681866    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 16:16:40.681897 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.681875    2559 server.go:87] "Starting device plugin registration server"
Jul 27 16:16:40.682125 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.682108    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 16:16:40.682228 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.682125    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 16:16:40.682299 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.682227    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 16:16:40.682359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.682336    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 16:16:40.682359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.682345    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 16:16:40.682953 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.682930    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 16:16:40.683048 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.682987    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.686134 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.686049    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-199.ec2.internal.18c631bf75ce797c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-199.ec2.internal,UID:ip-10-0-140-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-199.ec2.internal,},FirstTimestamp:2026-07-27 16:16:40.6839607 +0000 UTC m=+0.585101128,LastTimestamp:2026-07-27 16:16:40.6839607 +0000 UTC m=+0.585101128,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-199.ec2.internal,}"
Jul 27 16:16:40.704692 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.704668    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-199.ec2.internal\" not found" node="ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.728266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.728234    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 16:16:40.729486 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.729471    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 16:16:40.729565 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.729514    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 16:16:40.729565 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.729537    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 16:16:40.729653 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.729572    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 16:16:40.732541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.732516    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:16:40.782412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.782378    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.783359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.783341    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.783471 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.783378    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.783471 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.783394    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.783471 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.783426    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.795317 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.795299    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.795407 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.795323    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-199.ec2.internal\": node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.830338 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.830292    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"]
Jul 27 16:16:40.830477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.830380    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.831238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.831222    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.831346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.831257    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.831346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.831272    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.832641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.832625    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.832746 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.832731    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.832792 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.832775    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.833334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.833318    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.833402 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.833347    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.833402 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.833360    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.833402 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.833320    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.833488 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.833410    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.833488 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.833425    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.835173 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.835157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.835235 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.835183    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:40.835789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.835774    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:40.835789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.835795    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:40.835910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.835808    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:40.839708 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.839692    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.863026 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.863000    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-199.ec2.internal\" not found" node="ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.866414 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.866396    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-199.ec2.internal\" not found" node="ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.940743 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:40.940655    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:40.979524 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.979495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83df169d31731b8be500e8b8ca40ce24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-199.ec2.internal\" (UID: \"83df169d31731b8be500e8b8ca40ce24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.979644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.979524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74b4d3d8d322a7531b330f8ae0c2d07a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal\" (UID: \"74b4d3d8d322a7531b330f8ae0c2d07a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:40.979644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:40.979545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b4d3d8d322a7531b330f8ae0c2d07a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal\" (UID: \"74b4d3d8d322a7531b330f8ae0c2d07a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.041649 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:41.041601    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:41.079941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.079912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83df169d31731b8be500e8b8ca40ce24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-199.ec2.internal\" (UID: \"83df169d31731b8be500e8b8ca40ce24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.080024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.079946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74b4d3d8d322a7531b330f8ae0c2d07a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal\" (UID: \"74b4d3d8d322a7531b330f8ae0c2d07a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.080024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.079992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b4d3d8d322a7531b330f8ae0c2d07a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal\" (UID: \"74b4d3d8d322a7531b330f8ae0c2d07a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.080024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.080014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74b4d3d8d322a7531b330f8ae0c2d07a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal\" (UID: \"74b4d3d8d322a7531b330f8ae0c2d07a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.080132 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.080024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83df169d31731b8be500e8b8ca40ce24-config\") pod \"kube-apiserver-proxy-ip-10-0-140-199.ec2.internal\" (UID: \"83df169d31731b8be500e8b8ca40ce24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.080132 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.080028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/74b4d3d8d322a7531b330f8ae0c2d07a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal\" (UID: \"74b4d3d8d322a7531b330f8ae0c2d07a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.142103 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:41.142062    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:41.166244 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.166222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.170319 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.169904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.242794 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:41.242711    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:41.347328 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:41.347264    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:41.416242 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.416206    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 16:16:41.416733 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.416427    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 16:16:41.448016 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:41.447984    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:41.549030 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:41.548943    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-199.ec2.internal\" not found"
Jul 27 16:16:41.569418 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.569388    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:16:41.603675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.603649    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:16:41.604597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.604577    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:16:41.604688 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.604617    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:16:41.604688 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.604632    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:16:41.648637 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.648611    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:16:41.649924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.649896    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:11:40 +0000 UTC" deadline="2028-02-02 02:52:33.471065075 +0000 UTC"
Jul 27 16:16:41.649924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.649925    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13306h35m51.821144635s"
Jul 27 16:16:41.679768 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.679740    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.698151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.698124    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 16:16:41.700465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.700439    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal"
Jul 27 16:16:41.701744 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.701241    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:16:41.715629 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.715610    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 16:16:41.828430 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:41.828395    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b4d3d8d322a7531b330f8ae0c2d07a.slice/crio-c5a57d0afa9b883f5dd4e1f76e11e1c56caf87b8f185365ec707eecdde7d1f88 WatchSource:0}: Error finding container c5a57d0afa9b883f5dd4e1f76e11e1c56caf87b8f185365ec707eecdde7d1f88: Status 404 returned error can't find the container with id c5a57d0afa9b883f5dd4e1f76e11e1c56caf87b8f185365ec707eecdde7d1f88
Jul 27 16:16:41.832162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:41.832139    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:16:41.838185 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:41.838160    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83df169d31731b8be500e8b8ca40ce24.slice/crio-ae5d399d443ac8d2d09b744b4c9d79fab187e5f24f1cda5e972bcff3e293e3d5 WatchSource:0}: Error finding container ae5d399d443ac8d2d09b744b4c9d79fab187e5f24f1cda5e972bcff3e293e3d5: Status 404 returned error can't find the container with id ae5d399d443ac8d2d09b744b4c9d79fab187e5f24f1cda5e972bcff3e293e3d5
Jul 27 16:16:42.455695 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.455661    2559 apiserver.go:52] "Watching apiserver"
Jul 27 16:16:42.465529 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.465493    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 16:16:42.465962 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.465939    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-k4mgt","kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh","openshift-cluster-node-tuning-operator/tuned-nb4m9","openshift-dns/node-resolver-z2k8f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal","openshift-multus/multus-rs6h5","openshift-multus/network-metrics-daemon-nw2nf","kube-system/global-pull-secret-syncer-bbqvv","openshift-image-registry/node-ca-qmfdr","openshift-multus/multus-additional-cni-plugins-qfcd7","openshift-network-diagnostics/network-check-target-hblhv","openshift-network-operator/iptables-alerter-vzm7m","openshift-ovn-kubernetes/ovnkube-node-nc8mh"]
Jul 27 16:16:42.467833 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.467806    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.469057 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.469034    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.470453 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.470432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.472113 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.472093    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.472858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.472837    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 16:16:42.472858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.472858    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.473000 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.472838    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 16:16:42.473175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.473160    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.473452 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.473432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.473956 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.473937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v4tdq\""
Jul 27 16:16:42.474175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.474156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.474262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.474213    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.474935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.474712    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.474935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.474728    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5ftkm\""
Jul 27 16:16:42.475097 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.474953    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 16:16:42.475241 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.475153    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 16:16:42.475241 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.475166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.475372 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.475339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.475701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.475679    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5zjsh\""
Jul 27 16:16:42.476625 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.476600    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bfqjr\""
Jul 27 16:16:42.477133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.476880    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.477133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.477110    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.477272 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.477168    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.477272 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.477212    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.478057 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.478038    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7jc59\""
Jul 27 16:16:42.478203 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.478185    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 16:16:42.478383 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.478187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.478467 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.478443    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:42.478932 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.478911    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 16:16:42.478932 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.478935    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.479103 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.479074    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h57m8\""
Jul 27 16:16:42.479165 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.479109    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.479233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.479208    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 16:16:42.479995 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.479973    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 16:16:42.480091 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.480080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.481400 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.481354    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 16:16:42.483207 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.483011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:42.483207 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.483138    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:42.484766 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.484747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.485049 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.485025    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 16:16:42.485625 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.485603    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 16:16:42.485725 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.485675    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tvrb2\""
Jul 27 16:16:42.487024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.486598    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.487277 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487255    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-cni-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.487386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-etc-kubernetes\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.487386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487315    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-run-netns\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-netns\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.487541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-env-overrides\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysctl-d\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.487541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf283595-3a83-4674-af35-b7861330a709-hosts-file\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.487541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487497    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-ovn\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-socket-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.487541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf283595-3a83-4674-af35-b7861330a709-tmp-dir\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-cni-netd\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487579    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpm2m\" (UniqueName: \"kubernetes.io/projected/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-kube-api-access-vpm2m\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-var-lib-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487672    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b16964c-e80c-4655-b921-22e9d67dc359-agent-certs\") pod \"konnectivity-agent-k4mgt\" (UID: \"1b16964c-e80c-4655-b921-22e9d67dc359\") " pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysconfig\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-host\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-tmp\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b70654-56c0-4e61-9015-8e72e2200da2-host-slash\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-node-log\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-k8s-cni-cncf-io\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.487940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487925    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8w8np\""
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487931    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3357411f-3fd8-4a76-9d8d-e5af12e69251-kubelet-config\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-kubelet\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-log-socket\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.487955    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488105    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-cni-binary-copy\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-cni-multus\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-sys\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-lib-modules\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.488347    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzvw9\" (UniqueName: \"kubernetes.io/projected/bf283595-3a83-4674-af35-b7861330a709-kube-api-access-jzvw9\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-systemd\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovnkube-script-lib\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-registration-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-cnibin\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.488779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-conf-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-run\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-tuned\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsdn\" (UniqueName: \"kubernetes.io/projected/38b70654-56c0-4e61-9015-8e72e2200da2-kube-api-access-jnsdn\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-cni-bin\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-cni-bin\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-slash\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-socket-dir-parent\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-kubelet\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysctl-conf\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38b70654-56c0-4e61-9015-8e72e2200da2-iptables-alerter-script\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-etc-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.488978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-system-cni-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-sys-fs\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.489678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489060    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-systemd\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovnkube-config\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/4a45c8b2-ccda-4663-95ea-4f107ad566f6-kube-api-access-bfwnl\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3357411f-3fd8-4a76-9d8d-e5af12e69251-dbus\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b16964c-e80c-4655-b921-22e9d67dc359-konnectivity-ca\") pod \"konnectivity-agent-k4mgt\" (UID: \"1b16964c-e80c-4655-b921-22e9d67dc359\") " pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-device-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-modprobe-d\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-hostroot\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489328    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489347    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-multus-certs\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4zg\" (UniqueName: \"kubernetes.io/projected/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-kube-api-access-dd4zg\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-var-lib-kubelet\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovn-node-metrics-cert\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489419    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-systemd-units\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-os-release\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-daemon-config\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjpn\" (UniqueName: \"kubernetes.io/projected/3cd758ac-cb72-4041-b348-aa0e7846d9ca-kube-api-access-sbjpn\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.490468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.489491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-kubernetes\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.492665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.492642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 16:16:42.492853 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.492827    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nnr29\""
Jul 27 16:16:42.500738 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.500714    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 16:16:42.518410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.518325    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xvgt"
Jul 27 16:16:42.532038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.532017    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xvgt"
Jul 27 16:16:42.579625 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.579596    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 16:16:42.590656 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-kubelet\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.590818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-kubelet\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.590818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-log-socket\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.590818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtq7\" (UniqueName: \"kubernetes.io/projected/f795eac2-9fea-403e-ba0e-54c90483cc72-kube-api-access-9mtq7\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.590818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-cni-binary-copy\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-cni-multus\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590864    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-log-socket\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-sys\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-lib-modules\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzvw9\" (UniqueName: \"kubernetes.io/projected/bf283595-3a83-4674-af35-b7861330a709-kube-api-access-jzvw9\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-systemd\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-cni-multus\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.590991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-sys\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovnkube-script-lib\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-registration-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591061    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-systemd\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-lib-modules\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-cnibin\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591155    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-registration-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-conf-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-run\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-cnibin\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-tuned\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591259    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-conf-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsdn\" (UniqueName: \"kubernetes.io/projected/38b70654-56c0-4e61-9015-8e72e2200da2-kube-api-access-jnsdn\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-run\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-cni-bin\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.591582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-cni-bin\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-slash\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591604    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-cni-bin\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-cni-binary-copy\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-cni-bin\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591684    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-slash\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-os-release\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovnkube-script-lib\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-socket-dir-parent\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-kubelet\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysctl-conf\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-socket-dir-parent\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38b70654-56c0-4e61-9015-8e72e2200da2-iptables-alerter-script\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-etc-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.592397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-var-lib-kubelet\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-system-cni-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.591993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-etc-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-sys-fs\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysctl-conf\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-systemd\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-system-cni-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovnkube-config\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-sys-fs\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/4a45c8b2-ccda-4663-95ea-4f107ad566f6-kube-api-access-bfwnl\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-systemd\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3357411f-3fd8-4a76-9d8d-e5af12e69251-dbus\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b16964c-e80c-4655-b921-22e9d67dc359-konnectivity-ca\") pod \"konnectivity-agent-k4mgt\" (UID: \"1b16964c-e80c-4655-b921-22e9d67dc359\") " pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.593225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-device-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-modprobe-d\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-hostroot\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3357411f-3fd8-4a76-9d8d-e5af12e69251-dbus\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-multus-certs\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-hostroot\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-multus-certs\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-modprobe-d\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592449    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-device-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4zg\" (UniqueName: \"kubernetes.io/projected/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-kube-api-access-dd4zg\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-var-lib-kubelet\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38b70654-56c0-4e61-9015-8e72e2200da2-iptables-alerter-script\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovn-node-metrics-cert\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-systemd-units\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-system-cni-dir\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-systemd-units\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-cnibin\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.594117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592622    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovnkube-config\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-cni-binary-copy\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qntll\" (UniqueName: \"kubernetes.io/projected/e00e9260-9cdf-4b60-9185-121d37a15afa-kube-api-access-qntll\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-os-release\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-daemon-config\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjpn\" (UniqueName: \"kubernetes.io/projected/3cd758ac-cb72-4041-b348-aa0e7846d9ca-kube-api-access-sbjpn\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-kubernetes\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-var-lib-kubelet\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f795eac2-9fea-403e-ba0e-54c90483cc72-host\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-os-release\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f795eac2-9fea-403e-ba0e-54c90483cc72-serviceca\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1b16964c-e80c-4655-b921-22e9d67dc359-konnectivity-ca\") pod \"konnectivity-agent-k4mgt\" (UID: \"1b16964c-e80c-4655-b921-22e9d67dc359\") " pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-cni-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-kubernetes\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-etc-kubernetes\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-run-netns\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-netns\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.595087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.592995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-cni-dir\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-env-overrides\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-etc-kubernetes\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593064    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-netns\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-run-netns\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysctl-d\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf283595-3a83-4674-af35-b7861330a709-hosts-file\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-multus-daemon-config\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-ovn\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-socket-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf283595-3a83-4674-af35-b7861330a709-tmp-dir\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593290    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-cni-netd\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpm2m\" (UniqueName: \"kubernetes.io/projected/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-kube-api-access-vpm2m\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-var-lib-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a45c8b2-ccda-4663-95ea-4f107ad566f6-env-overrides\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593438    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-socket-dir\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b16964c-e80c-4655-b921-22e9d67dc359-agent-certs\") pod \"konnectivity-agent-k4mgt\" (UID: \"1b16964c-e80c-4655-b921-22e9d67dc359\") " pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysctl-d\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68xv\" (UniqueName: \"kubernetes.io/projected/6f2d7be9-b333-4690-97ec-820cc701173b-kube-api-access-g68xv\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysconfig\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bf283595-3a83-4674-af35-b7861330a709-hosts-file\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-cni-netd\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-host\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-tmp\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.593582    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.593687    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:16:43.093664161 +0000 UTC m=+2.994804607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-var-lib-openvswitch\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.593934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-sysconfig\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-run-ovn\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf283595-3a83-4674-af35-b7861330a709-tmp-dir\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-host\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.596834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b70654-56c0-4e61-9015-8e72e2200da2-host-slash\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-node-log\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-k8s-cni-cncf-io\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3357411f-3fd8-4a76-9d8d-e5af12e69251-kubelet-config\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38b70654-56c0-4e61-9015-8e72e2200da2-host-slash\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a45c8b2-ccda-4663-95ea-4f107ad566f6-node-log\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-host-run-k8s-cni-cncf-io\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594937    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cd758ac-cb72-4041-b348-aa0e7846d9ca-etc-selinux\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.594983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3357411f-3fd8-4a76-9d8d-e5af12e69251-kubelet-config\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.596334    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-etc-tuned\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.596459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a45c8b2-ccda-4663-95ea-4f107ad566f6-ovn-node-metrics-cert\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.597017    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1b16964c-e80c-4655-b921-22e9d67dc359-agent-certs\") pod \"konnectivity-agent-k4mgt\" (UID: \"1b16964c-e80c-4655-b921-22e9d67dc359\") " pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.597548 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.597205    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-tmp\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.600800 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.600776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzvw9\" (UniqueName: \"kubernetes.io/projected/bf283595-3a83-4674-af35-b7861330a709-kube-api-access-jzvw9\") pod \"node-resolver-z2k8f\" (UID: \"bf283595-3a83-4674-af35-b7861330a709\") " pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.604228 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.604202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwnl\" (UniqueName: \"kubernetes.io/projected/4a45c8b2-ccda-4663-95ea-4f107ad566f6-kube-api-access-bfwnl\") pod \"ovnkube-node-nc8mh\" (UID: \"4a45c8b2-ccda-4663-95ea-4f107ad566f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.604387 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.604362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsdn\" (UniqueName: \"kubernetes.io/projected/38b70654-56c0-4e61-9015-8e72e2200da2-kube-api-access-jnsdn\") pod \"iptables-alerter-vzm7m\" (UID: \"38b70654-56c0-4e61-9015-8e72e2200da2\") " pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.604772 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.604751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjpn\" (UniqueName: \"kubernetes.io/projected/3cd758ac-cb72-4041-b348-aa0e7846d9ca-kube-api-access-sbjpn\") pod \"aws-ebs-csi-driver-node-727qh\" (UID: \"3cd758ac-cb72-4041-b348-aa0e7846d9ca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.606938 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.606914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4zg\" (UniqueName: \"kubernetes.io/projected/5758b2ba-b3c7-4612-a2fd-89e46b108cf7-kube-api-access-dd4zg\") pod \"multus-rs6h5\" (UID: \"5758b2ba-b3c7-4612-a2fd-89e46b108cf7\") " pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.608687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.608665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpm2m\" (UniqueName: \"kubernetes.io/projected/2d51eb4d-c3d7-4a7e-883e-34a53347c11c-kube-api-access-vpm2m\") pod \"tuned-nb4m9\" (UID: \"2d51eb4d-c3d7-4a7e-883e-34a53347c11c\") " pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.625156 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.625133    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:16:42.695038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.694999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtq7\" (UniqueName: \"kubernetes.io/projected/f795eac2-9fea-403e-ba0e-54c90483cc72-kube-api-access-9mtq7\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.695038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-os-release\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-system-cni-dir\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695198    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-cnibin\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-cni-binary-copy\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.695231    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695252    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qntll\" (UniqueName: \"kubernetes.io/projected/e00e9260-9cdf-4b60-9185-121d37a15afa-kube-api-access-qntll\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695281 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f795eac2-9fea-403e-ba0e-54c90483cc72-host\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f795eac2-9fea-403e-ba0e-54c90483cc72-serviceca\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.695410    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:16:43.195392907 +0000 UTC m=+3.096533343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f795eac2-9fea-403e-ba0e-54c90483cc72-host\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-system-cni-dir\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-cnibin\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e00e9260-9cdf-4b60-9185-121d37a15afa-os-release\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.695653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68xv\" (UniqueName: \"kubernetes.io/projected/6f2d7be9-b333-4690-97ec-820cc701173b-kube-api-access-g68xv\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:42.695966 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.695806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f795eac2-9fea-403e-ba0e-54c90483cc72-serviceca\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.696283 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.696262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.696389 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.696289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-cni-binary-copy\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.696457 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.696436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e00e9260-9cdf-4b60-9185-121d37a15afa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.703005 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.702968    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:16:42.703005 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.702994    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:16:42.703005 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.703005    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:42.703219 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:42.703097    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:16:43.203079606 +0000 UTC m=+3.104220041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:42.704840 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.704812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtq7\" (UniqueName: \"kubernetes.io/projected/f795eac2-9fea-403e-ba0e-54c90483cc72-kube-api-access-9mtq7\") pod \"node-ca-qmfdr\" (UID: \"f795eac2-9fea-403e-ba0e-54c90483cc72\") " pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.705693 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.705676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qntll\" (UniqueName: \"kubernetes.io/projected/e00e9260-9cdf-4b60-9185-121d37a15afa-kube-api-access-qntll\") pod \"multus-additional-cni-plugins-qfcd7\" (UID: \"e00e9260-9cdf-4b60-9185-121d37a15afa\") " pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:42.710864 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.710804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68xv\" (UniqueName: \"kubernetes.io/projected/6f2d7be9-b333-4690-97ec-820cc701173b-kube-api-access-g68xv\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:42.734691 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.734641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal" event={"ID":"83df169d31731b8be500e8b8ca40ce24","Type":"ContainerStarted","Data":"ae5d399d443ac8d2d09b744b4c9d79fab187e5f24f1cda5e972bcff3e293e3d5"}
Jul 27 16:16:42.735958 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.735930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal" event={"ID":"74b4d3d8d322a7531b330f8ae0c2d07a","Type":"ContainerStarted","Data":"c5a57d0afa9b883f5dd4e1f76e11e1c56caf87b8f185365ec707eecdde7d1f88"}
Jul 27 16:16:42.781628 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.781602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rs6h5"
Jul 27 16:16:42.790802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.790777    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vzm7m"
Jul 27 16:16:42.799758 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.799732    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh"
Jul 27 16:16:42.805934 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.805914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nb4m9"
Jul 27 16:16:42.814636 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.814611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2k8f"
Jul 27 16:16:42.820443 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.820423    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:16:42.827435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.827413    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:16:42.834448 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.834426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qmfdr"
Jul 27 16:16:42.841168 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:42.841149    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qfcd7"
Jul 27 16:16:43.015950 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.015923    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd758ac_cb72_4041_b348_aa0e7846d9ca.slice/crio-d80f27c10cd453a24ddedaf5997ae5071a20c46facc85d3f51fd3ad361ff68a5 WatchSource:0}: Error finding container d80f27c10cd453a24ddedaf5997ae5071a20c46facc85d3f51fd3ad361ff68a5: Status 404 returned error can't find the container with id d80f27c10cd453a24ddedaf5997ae5071a20c46facc85d3f51fd3ad361ff68a5
Jul 27 16:16:43.018078 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.018054    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a45c8b2_ccda_4663_95ea_4f107ad566f6.slice/crio-5eb87a1ddd861ac36240bb6c7454e342d34204c25753471b984566591d521a86 WatchSource:0}: Error finding container 5eb87a1ddd861ac36240bb6c7454e342d34204c25753471b984566591d521a86: Status 404 returned error can't find the container with id 5eb87a1ddd861ac36240bb6c7454e342d34204c25753471b984566591d521a86
Jul 27 16:16:43.033338 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.033274    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5758b2ba_b3c7_4612_a2fd_89e46b108cf7.slice/crio-ff52b4b4a3aa67a7a73646e37d3ca79e8cb672e3ad72144cc7177585b93cda16 WatchSource:0}: Error finding container ff52b4b4a3aa67a7a73646e37d3ca79e8cb672e3ad72144cc7177585b93cda16: Status 404 returned error can't find the container with id ff52b4b4a3aa67a7a73646e37d3ca79e8cb672e3ad72144cc7177585b93cda16
Jul 27 16:16:43.035759 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.035734    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b16964c_e80c_4655_b921_22e9d67dc359.slice/crio-80c1ad4ee3d898a6be7c2a45f9b5c5b20fc640e86556f77a56f24728f0acbcba WatchSource:0}: Error finding container 80c1ad4ee3d898a6be7c2a45f9b5c5b20fc640e86556f77a56f24728f0acbcba: Status 404 returned error can't find the container with id 80c1ad4ee3d898a6be7c2a45f9b5c5b20fc640e86556f77a56f24728f0acbcba
Jul 27 16:16:43.040074 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.039994    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b70654_56c0_4e61_9015_8e72e2200da2.slice/crio-5e826ba0f598b4d088787bcff4c1f73d4516189127a272a20b7b68af674cb6dc WatchSource:0}: Error finding container 5e826ba0f598b4d088787bcff4c1f73d4516189127a272a20b7b68af674cb6dc: Status 404 returned error can't find the container with id 5e826ba0f598b4d088787bcff4c1f73d4516189127a272a20b7b68af674cb6dc
Jul 27 16:16:43.040430 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.040406    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d51eb4d_c3d7_4a7e_883e_34a53347c11c.slice/crio-3d5363ecfc2ea209ecf9dbe961a6cbfbe0ef24e681c0d4e64b6347d8d6011be4 WatchSource:0}: Error finding container 3d5363ecfc2ea209ecf9dbe961a6cbfbe0ef24e681c0d4e64b6347d8d6011be4: Status 404 returned error can't find the container with id 3d5363ecfc2ea209ecf9dbe961a6cbfbe0ef24e681c0d4e64b6347d8d6011be4
Jul 27 16:16:43.040993 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.040881    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf795eac2_9fea_403e_ba0e_54c90483cc72.slice/crio-d44fc6feeff53432aff79be47e8dae7f24b62bc3f661a7fd4c85288aed1a7bbd WatchSource:0}: Error finding container d44fc6feeff53432aff79be47e8dae7f24b62bc3f661a7fd4c85288aed1a7bbd: Status 404 returned error can't find the container with id d44fc6feeff53432aff79be47e8dae7f24b62bc3f661a7fd4c85288aed1a7bbd
Jul 27 16:16:43.042895 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.042595    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00e9260_9cdf_4b60_9185_121d37a15afa.slice/crio-ca6e137c28bb61842e462a59c0b306c873f8f6f6bfa934da2e4365c998968204 WatchSource:0}: Error finding container ca6e137c28bb61842e462a59c0b306c873f8f6f6bfa934da2e4365c998968204: Status 404 returned error can't find the container with id ca6e137c28bb61842e462a59c0b306c873f8f6f6bfa934da2e4365c998968204
Jul 27 16:16:43.042895 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:16:43.042761    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf283595_3a83_4674_af35_b7861330a709.slice/crio-2e6274f650ef27e083d7367717a4a7cb9671b4af26b6095ebdc855d2f7d77fc7 WatchSource:0}: Error finding container 2e6274f650ef27e083d7367717a4a7cb9671b4af26b6095ebdc855d2f7d77fc7: Status 404 returned error can't find the container with id 2e6274f650ef27e083d7367717a4a7cb9671b4af26b6095ebdc855d2f7d77fc7
Jul 27 16:16:43.099177 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.099142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:43.099335 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.099299    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:43.099390 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.099362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:16:44.099346948 +0000 UTC m=+4.000487375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:43.199647 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.199605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:43.199801 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.199751    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:43.199881 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.199831    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:16:44.199811414 +0000 UTC m=+4.100951854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:43.300776 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.300745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:43.300902 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.300886    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:16:43.300947 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.300911    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:16:43.300947 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.300922    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:43.301016 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.300976    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:16:44.300959764 +0000 UTC m=+4.202100207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:43.534904 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.534799    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:11:42 +0000 UTC" deadline="2028-03-23 06:10:08.278233099 +0000 UTC"
Jul 27 16:16:43.534904 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.534843    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14509h53m24.743394981s"
Jul 27 16:16:43.731442 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.730836    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:43.731442 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.730864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:43.731442 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.731011    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:43.731442 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:43.731153    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:43.751473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.751439    2559 generic.go:356] "Generic (PLEG): container finished" podID="74b4d3d8d322a7531b330f8ae0c2d07a" containerID="6d3dd3f814b3c20ce082c14ce4fb184e0b033c0206fcb5c679f42ba23daa17c3" exitCode=0
Jul 27 16:16:43.751754 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.751728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal" event={"ID":"74b4d3d8d322a7531b330f8ae0c2d07a","Type":"ContainerDied","Data":"6d3dd3f814b3c20ce082c14ce4fb184e0b033c0206fcb5c679f42ba23daa17c3"}
Jul 27 16:16:43.760055 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.759995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerStarted","Data":"ca6e137c28bb61842e462a59c0b306c873f8f6f6bfa934da2e4365c998968204"}
Jul 27 16:16:43.767240 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.767174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qmfdr" event={"ID":"f795eac2-9fea-403e-ba0e-54c90483cc72","Type":"ContainerStarted","Data":"d44fc6feeff53432aff79be47e8dae7f24b62bc3f661a7fd4c85288aed1a7bbd"}
Jul 27 16:16:43.770062 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.769797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nb4m9" event={"ID":"2d51eb4d-c3d7-4a7e-883e-34a53347c11c","Type":"ContainerStarted","Data":"3d5363ecfc2ea209ecf9dbe961a6cbfbe0ef24e681c0d4e64b6347d8d6011be4"}
Jul 27 16:16:43.788229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.788115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4mgt" event={"ID":"1b16964c-e80c-4655-b921-22e9d67dc359","Type":"ContainerStarted","Data":"80c1ad4ee3d898a6be7c2a45f9b5c5b20fc640e86556f77a56f24728f0acbcba"}
Jul 27 16:16:43.803050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.803020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"5eb87a1ddd861ac36240bb6c7454e342d34204c25753471b984566591d521a86"}
Jul 27 16:16:43.806394 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.806368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2k8f" event={"ID":"bf283595-3a83-4674-af35-b7861330a709","Type":"ContainerStarted","Data":"2e6274f650ef27e083d7367717a4a7cb9671b4af26b6095ebdc855d2f7d77fc7"}
Jul 27 16:16:43.815611 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.815562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vzm7m" event={"ID":"38b70654-56c0-4e61-9015-8e72e2200da2","Type":"ContainerStarted","Data":"5e826ba0f598b4d088787bcff4c1f73d4516189127a272a20b7b68af674cb6dc"}
Jul 27 16:16:43.821174 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.821096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rs6h5" event={"ID":"5758b2ba-b3c7-4612-a2fd-89e46b108cf7","Type":"ContainerStarted","Data":"ff52b4b4a3aa67a7a73646e37d3ca79e8cb672e3ad72144cc7177585b93cda16"}
Jul 27 16:16:43.828041 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:43.827937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh" event={"ID":"3cd758ac-cb72-4041-b348-aa0e7846d9ca","Type":"ContainerStarted","Data":"d80f27c10cd453a24ddedaf5997ae5071a20c46facc85d3f51fd3ad361ff68a5"}
Jul 27 16:16:44.110718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.110632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:44.110908 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.110828    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:44.110908 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.110896    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:16:46.110875882 +0000 UTC m=+6.012016315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:44.211839 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.211791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:44.212026 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.211956    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:44.212026 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.212021    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:16:46.212002399 +0000 UTC m=+6.113142829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:44.313060 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.313022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:44.313246 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.313207    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:16:44.313246 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.313228    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:16:44.313246 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.313241    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:44.313441 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.313307    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:16:46.313286229 +0000 UTC m=+6.214426674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:44.535672 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.535553    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:11:42 +0000 UTC" deadline="2027-12-21 14:20:08.487664314 +0000 UTC"
Jul 27 16:16:44.535672 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.535622    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12286h3m23.952047327s"
Jul 27 16:16:44.737478 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.736967    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:44.737478 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:44.737108    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:44.850837 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:44.850713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal" event={"ID":"74b4d3d8d322a7531b330f8ae0c2d07a","Type":"ContainerStarted","Data":"e472ffaeeb636072c1230e7f22068f8130af8f4fa5b8388af27c85bac7665e84"}
Jul 27 16:16:45.731519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:45.730714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:45.731519 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:45.730852    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:45.731519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:45.731247    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:45.731519 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:45.731366    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:46.128213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:46.128120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:46.128424 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.128284    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:46.128424 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.128371    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:16:50.128352016 +0000 UTC m=+10.029492446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:46.229356 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:46.229316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:46.229539 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.229501    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:46.229588 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.229572    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:16:50.229549108 +0000 UTC m=+10.130689553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:46.330315 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:46.329666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:46.330315 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.329867    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:16:46.330315 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.329891    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:16:46.330315 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.329905    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:46.330315 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.329961    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:16:50.329941729 +0000 UTC m=+10.231082192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:46.730484 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:46.730444    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:46.730656 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:46.730579    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:47.730011 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:47.729972    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:47.730473 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:47.730120    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:47.730473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:47.730157    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:47.730473 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:47.730268    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:48.732456 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:48.732421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:48.732923 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:48.732574    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:49.730748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:49.730041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:49.730748 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:49.730172    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:49.730748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:49.730237    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:49.730748 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:49.730324    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:50.160122 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:50.160037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:50.160556 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.160218    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:50.160556 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.160295    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:16:58.160264271 +0000 UTC m=+18.061404696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:50.260701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:50.260655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:50.260863 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.260819    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:50.260936 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.260893    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:16:58.260873893 +0000 UTC m=+18.162014319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:50.362669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:50.362054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:50.362669 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.362216    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:16:50.362669 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.362239    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:16:50.362669 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.362253    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:50.362669 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.362327    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:16:58.36230952 +0000 UTC m=+18.263449949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:50.731998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:50.731433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:50.731998 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:50.731573    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:51.730434 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:51.730391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:51.730905 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:51.730526    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:51.730905 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:51.730572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:51.730905 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:51.730661    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:52.730801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:52.730772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:52.731354 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:52.730913    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:53.729758 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:53.729725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:53.729903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:53.729725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:53.729903 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:53.729832    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:53.729995 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:53.729900    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:54.730878 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:54.730838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:54.731348 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:54.730981    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:55.730110 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:55.730076    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:55.730275 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:55.730076    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:55.730275 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:55.730218    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:55.730413 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:55.730294    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:56.730110 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:56.730075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:56.730598 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:56.730223    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:57.729718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:57.729680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:57.729718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:57.729709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:57.729964 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:57.729791    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:16:57.729964 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:57.729899    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:58.218726 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:58.218641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:58.219143 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.218764    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:58.219143 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.218825    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:14.218807321 +0000 UTC m=+34.119947751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:16:58.319866 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:58.319827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:58.320061 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.319993    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:58.320124 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.320073    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:14.320050129 +0000 UTC m=+34.221190586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:16:58.421045 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:58.421009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:58.421211 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.421159    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:16:58.421211 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.421185    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:16:58.421211 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.421199    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:58.421392 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.421255    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:17:14.421237055 +0000 UTC m=+34.322377481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:16:58.730113 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:58.730086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:16:58.730307 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:58.730213    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:16:59.729753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:59.729716    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:16:59.729753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:16:59.729754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:16:59.730197 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:59.729857    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:16:59.730197 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:16:59.729979    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:00.730740 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:00.730699    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:00.731174 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:00.730835    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:01.730399 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:01.730363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:01.730538 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:01.730480    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:01.730600 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:01.730534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:01.730674 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:01.730654    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:02.730861 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.730587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:02.731449 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:02.730975    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:02.889357 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.889329    2559 generic.go:356] "Generic (PLEG): container finished" podID="e00e9260-9cdf-4b60-9185-121d37a15afa" containerID="5a3342756e8f5ddc63de86017d355b95ee31b31b3bfcc6f274001a7e49e8e15e" exitCode=0
Jul 27 16:17:02.889514 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.889411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerDied","Data":"5a3342756e8f5ddc63de86017d355b95ee31b31b3bfcc6f274001a7e49e8e15e"}
Jul 27 16:17:02.890983 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.890960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qmfdr" event={"ID":"f795eac2-9fea-403e-ba0e-54c90483cc72","Type":"ContainerStarted","Data":"d6cd4a69d3af85344fdc30d785ce8bdf7f3d8e82ecb02e3bc170269668365467"}
Jul 27 16:17:02.892340 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.892318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nb4m9" event={"ID":"2d51eb4d-c3d7-4a7e-883e-34a53347c11c","Type":"ContainerStarted","Data":"f00f5f569722e46aecd3d6ae3c7b2e1bfcfa7109efec48cc7c2171d3e0635e31"}
Jul 27 16:17:02.893553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.893530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k4mgt" event={"ID":"1b16964c-e80c-4655-b921-22e9d67dc359","Type":"ContainerStarted","Data":"c01eeb536dbc1fea0ad7666e9c25af5c8e690fd3a57857783b75daa56ae3c1a8"}
Jul 27 16:17:02.896435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.896416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"108c46710b6de70ac254af9b6e362d171ff7c18acc94f24c8cafe7d701ec6b0a"}
Jul 27 16:17:02.896527 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.896475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"ed88dbb8a9b353b81b09c2a74ed8ebbbbf18abccc1dd3718ccbe6857f96f1ec9"}
Jul 27 16:17:02.896527 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.896521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"04b3c6cb21b88982a464d931767856902dc3597d6896b928cc0182a9a1031497"}
Jul 27 16:17:02.896723 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.896566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"24dfa8543d4d5efe6bbe89be6a79ff6e074ab3c944f699e69a300c62c002194c"}
Jul 27 16:17:02.896723 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.896599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"b8b929e0ef44fe40f441c8bfc670b0075f1fdd3381d709860d91aafe18569c3b"}
Jul 27 16:17:02.897927 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.897910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal" event={"ID":"83df169d31731b8be500e8b8ca40ce24","Type":"ContainerStarted","Data":"ff1147190da140946d6518a95d348879092e3136d551331c9e7bd4200913f432"}
Jul 27 16:17:02.899325 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.899304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2k8f" event={"ID":"bf283595-3a83-4674-af35-b7861330a709","Type":"ContainerStarted","Data":"958da1b540967efb541b44f9c8a4c35d186052962a4dd21aa5be64820f1c479c"}
Jul 27 16:17:02.900681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.900661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rs6h5" event={"ID":"5758b2ba-b3c7-4612-a2fd-89e46b108cf7","Type":"ContainerStarted","Data":"a5b27c38245de6c6abf75c5730119d05253b30cc49f91e4fe66622833e67d09b"}
Jul 27 16:17:02.902079 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.902058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh" event={"ID":"3cd758ac-cb72-4041-b348-aa0e7846d9ca","Type":"ContainerStarted","Data":"cfe3de00096d34db57691dfbeea07d56c1bdf3ce27f32fd98ab7a1d9029e3f22"}
Jul 27 16:17:02.966271 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:02.966211    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-199.ec2.internal" podStartSLOduration=20.96619768 podStartE2EDuration="20.96619768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:16:44.877791868 +0000 UTC m=+4.778932313" watchObservedRunningTime="2026-07-27 16:17:02.96619768 +0000 UTC m=+22.867338127"
Jul 27 16:17:03.007334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.007270    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qmfdr" podStartSLOduration=7.312004525 podStartE2EDuration="22.007254721s" totalImagesPullingTime="14.695250196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.006849019 +0000 UTC m=+22.907989467" watchObservedRunningTime="2026-07-27 16:17:03.007254721 +0000 UTC m=+22.908395169"
Jul 27 16:17:03.040445 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.040375    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rs6h5" podStartSLOduration=2.709891266 podStartE2EDuration="22.040354396s" totalImagesPullingTime="19.33046313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.039656489 +0000 UTC m=+22.940796939" watchObservedRunningTime="2026-07-27 16:17:03.040354396 +0000 UTC m=+22.941494848"
Jul 27 16:17:03.069164 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.069104    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nb4m9" podStartSLOduration=2.743425832 podStartE2EDuration="22.069082177s" totalImagesPullingTime="19.325656345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.068237465 +0000 UTC m=+22.969377936" watchObservedRunningTime="2026-07-27 16:17:03.069082177 +0000 UTC m=+22.970222626"
Jul 27 16:17:03.129226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.129125    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k4mgt" podStartSLOduration=3.482207775 podStartE2EDuration="22.129109719s" totalImagesPullingTime="18.646901944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.128766419 +0000 UTC m=+23.029906870" watchObservedRunningTime="2026-07-27 16:17:03.129109719 +0000 UTC m=+23.030250169"
Jul 27 16:17:03.129512 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.129468    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z2k8f" podStartSLOduration=2.8058555849999998 podStartE2EDuration="22.129457044s" totalImagesPullingTime="19.323601459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.094562419 +0000 UTC m=+22.995702868" watchObservedRunningTime="2026-07-27 16:17:03.129457044 +0000 UTC m=+23.030597490"
Jul 27 16:17:03.171233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.171172    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-199.ec2.internal" podStartSLOduration=21.171150077 podStartE2EDuration="21.171150077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.169903104 +0000 UTC m=+23.071043563" watchObservedRunningTime="2026-07-27 16:17:03.171150077 +0000 UTC m=+23.072290526"
Jul 27 16:17:03.730607 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.730582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:03.730751 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:03.730673    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:03.730751 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.730724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:03.730822 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:03.730805    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:03.851766 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.851743    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 16:17:03.906500 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.906425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"fdd5bdb6cca7ab65c094c52f9c1afe09e8092b01ea37b510c2875a2836241146"}
Jul 27 16:17:03.908229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.908196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vzm7m" event={"ID":"38b70654-56c0-4e61-9015-8e72e2200da2","Type":"ContainerStarted","Data":"205343b0481ebaa3f982172939aa16ba3eede7b015725e839d25664cef59ba00"}
Jul 27 16:17:03.911114 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.911067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh" event={"ID":"3cd758ac-cb72-4041-b348-aa0e7846d9ca","Type":"ContainerStarted","Data":"ac70c99f3b65b6a85a1e37aa1eaa328e47e8b0f625aa197dc2e90c798c1c194a"}
Jul 27 16:17:03.927051 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.927009    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vzm7m" podStartSLOduration=3.6023368749999998 podStartE2EDuration="22.926996747s" totalImagesPullingTime="19.324659872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:03.925865598 +0000 UTC m=+23.827006045" watchObservedRunningTime="2026-07-27 16:17:03.926996747 +0000 UTC m=+23.828137195"
Jul 27 16:17:03.956827 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.956799    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:17:03.957802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:03.957782    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:17:04.699691 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.699576    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T16:17:03.851761437Z","UUID":"fcbd2365-a30a-432c-9ae6-312e4feb31aa","Handler":null,"Name":"","Endpoint":""}
Jul 27 16:17:04.701339 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.701317    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 16:17:04.701444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.701365    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 16:17:04.730364 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.730343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:04.730497 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:04.730479    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:04.915853 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.915761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh" event={"ID":"3cd758ac-cb72-4041-b348-aa0e7846d9ca","Type":"ContainerStarted","Data":"40799b839ec30c320dc38e3446428a93e2e57d454756b0ec282ee561ab509060"}
Jul 27 16:17:04.916322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.916239    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:17:04.916519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.916505    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k4mgt"
Jul 27 16:17:04.957555 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:04.957496    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-727qh" podStartSLOduration=2.734377845 podStartE2EDuration="23.95748037s" totalImagesPullingTime="21.223102525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:04.939101799 +0000 UTC m=+24.840242246" watchObservedRunningTime="2026-07-27 16:17:04.95748037 +0000 UTC m=+24.858620821"
Jul 27 16:17:05.729877 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:05.729847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:05.730038 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:05.729947    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:05.730038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:05.729999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:05.730110 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:05.730077    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:05.921721 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:05.921535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"9ef2f799333db8ed49f03bd998ad6bf9b34c32a7dab20810d58413bb351f1567"}
Jul 27 16:17:06.732856 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:06.732831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:06.733014 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:06.732934    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:07.729953 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:07.729917    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:07.730733 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:07.730030    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:07.730733 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:07.730081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:07.730733 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:07.730173    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:07.928427 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:07.928389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" event={"ID":"4a45c8b2-ccda-4663-95ea-4f107ad566f6","Type":"ContainerStarted","Data":"23cc1bc775e6c33e1b33143d69590bd3788f0a058ddc85c2328afa6c473b6974"}
Jul 27 16:17:07.928641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:07.928625    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:17:07.943923 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:07.943892    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:17:07.961605 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:07.961564    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh" podStartSLOduration=7.624804585 podStartE2EDuration="26.961550894s" totalImagesPullingTime="19.336746309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:07.961240358 +0000 UTC m=+27.862380846" watchObservedRunningTime="2026-07-27 16:17:07.961550894 +0000 UTC m=+27.862691341"
Jul 27 16:17:08.730070 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:08.730035    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:08.730553 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:08.730176    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:08.931479 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:08.931444    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:17:08.931653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:08.931524    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:17:08.947975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:08.947816    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:17:09.420076 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:09.419995    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbqvv"]
Jul 27 16:17:09.420209 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:09.420131    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:09.420275 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:09.420251    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:09.436491 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:09.436452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nw2nf"]
Jul 27 16:17:09.436641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:09.436568    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:09.436706 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:09.436688    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:09.436970 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:09.436949    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hblhv"]
Jul 27 16:17:09.437050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:09.437039    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:09.437129 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:09.437114    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:10.736991 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:10.736962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:10.737436 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:10.736963    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:10.737436 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:10.737102    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:10.737436 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:10.737192    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:11.730143 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:11.730118    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:11.730386 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:11.730212    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:12.733449 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:12.733421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:12.733804 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:12.733461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:12.733804 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:12.733522    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:12.733804 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:12.733625    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:13.730467 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:13.730437    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:13.730682 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:13.730534    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hblhv" podUID="d55354b4-55ba-4fed-bab5-7035641c759e"
Jul 27 16:17:14.239627 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.239591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:14.240013 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.239756    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:14.240013 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.239826    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret podName:3357411f-3fd8-4a76-9d8d-e5af12e69251 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:46.239806894 +0000 UTC m=+66.140947321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret") pod "global-pull-secret-syncer-bbqvv" (UID: "3357411f-3fd8-4a76-9d8d-e5af12e69251") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:14.340589 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.340550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:14.340750 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.340670    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:14.340750 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.340732    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:46.340716451 +0000 UTC m=+66.241856877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:14.441372 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.441337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:14.441530 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.441469    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:14.441530 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.441484    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:14.441530 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.441493    2559 projected.go:196] Error preparing data for projected volume kube-api-access-dgzj5 for pod openshift-network-diagnostics/network-check-target-hblhv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:14.441642 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.441542    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5 podName:d55354b4-55ba-4fed-bab5-7035641c759e nodeName:}" failed. No retries permitted until 2026-07-27 16:17:46.441526446 +0000 UTC m=+66.342666873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dgzj5" (UniqueName: "kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5") pod "network-check-target-hblhv" (UID: "d55354b4-55ba-4fed-bab5-7035641c759e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:14.733238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.733203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:14.733438 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.733342    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nw2nf" podUID="6f2d7be9-b333-4690-97ec-820cc701173b"
Jul 27 16:17:14.733438 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.733389    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:14.733549 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:14.733488    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bbqvv" podUID="3357411f-3fd8-4a76-9d8d-e5af12e69251"
Jul 27 16:17:14.968079 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.968051    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-199.ec2.internal" event="NodeReady"
Jul 27 16:17:14.968216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:14.968164    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 16:17:15.019601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.019535    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"]
Jul 27 16:17:15.025660 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.025619    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"]
Jul 27 16:17:15.025814 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.025768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.028275 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.028254    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lrjb8\""
Jul 27 16:17:15.029054 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.029028    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 16:17:15.029162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.029031    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 16:17:15.029162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.029063    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 16:17:15.029162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.029105    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 16:17:15.030766 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.030749    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"]
Jul 27 16:17:15.030885 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.030869    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.033835 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.033814    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c69f7688b-52vqs"]
Jul 27 16:17:15.033945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.033920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.034351 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.034077    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 16:17:15.034351 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.034336    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 16:17:15.035061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.034858    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 16:17:15.035061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.034913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 16:17:15.036602 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.036585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"]
Jul 27 16:17:15.036719 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.036659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.037641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.037621    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 16:17:15.038675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.038658    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"]
Jul 27 16:17:15.039109 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.039084    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 16:17:15.039215 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.039161    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9k8c5\""
Jul 27 16:17:15.040074 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.040055    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 16:17:15.040149 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.040092    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 16:17:15.042997 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.042977    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pcfdm"]
Jul 27 16:17:15.047054 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.047030    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kmzm2"]
Jul 27 16:17:15.047169 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.047158    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.050499 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.050462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c69f7688b-52vqs"]
Jul 27 16:17:15.050612 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.050565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.050695 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.050617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 16:17:15.051003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.050984    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 16:17:15.051106 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.050986    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 16:17:15.051106 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.051039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2mnq\""
Jul 27 16:17:15.053038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.052950    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"]
Jul 27 16:17:15.053801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.053783    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 16:17:15.054101 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.054078    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 16:17:15.054560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.054541    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 16:17:15.054640 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.054625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9rtzs\""
Jul 27 16:17:15.061111 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.061092    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmzm2"]
Jul 27 16:17:15.066344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.066322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcfdm"]
Jul 27 16:17:15.146741 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e47df4f-c390-4f91-960b-45f793be4e91-tmp\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.146902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tk5\" (UniqueName: \"kubernetes.io/projected/a3d180c6-053f-43a8-8d83-f8f714ea5907-kube-api-access-86tk5\") pod \"managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6\" (UID: \"a3d180c6-053f-43a8-8d83-f8f714ea5907\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.146902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.146902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grlsz\" (UniqueName: \"kubernetes.io/projected/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-kube-api-access-grlsz\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.146902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3d01532-a5ff-46fb-b8d2-dd8117802655-ca-trust-extracted\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.146902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-hub\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7e47df4f-c390-4f91-960b-45f793be4e91-klusterlet-config\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgfl\" (UniqueName: \"kubernetes.io/projected/cd1af325-26fe-4a63-8de9-f7ebe2f189a9-kube-api-access-rtgfl\") pod \"ingress-canary-kmzm2\" (UID: \"cd1af325-26fe-4a63-8de9-f7ebe2f189a9\") " pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.146957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1af325-26fe-4a63-8de9-f7ebe2f189a9-cert\") pod \"ingress-canary-kmzm2\" (UID: \"cd1af325-26fe-4a63-8de9-f7ebe2f189a9\") " pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3d180c6-053f-43a8-8d83-f8f714ea5907-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6\" (UID: \"a3d180c6-053f-43a8-8d83-f8f714ea5907\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4dp\" (UniqueName: \"kubernetes.io/projected/7e47df4f-c390-4f91-960b-45f793be4e91-kube-api-access-8t4dp\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-tmp-dir\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-installation-pull-secrets\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.147094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147095    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4jr\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-kube-api-access-gw4jr\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147148    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f697167f-d912-464a-a29b-48cdd79a9154-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147175    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-config-volume\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-bound-sa-token\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-ca\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-image-registry-private-configuration\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-trusted-ca\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2k2p\" (UniqueName: \"kubernetes.io/projected/f697167f-d912-464a-a29b-48cdd79a9154-kube-api-access-n2k2p\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147354    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.147519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.147378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-certificates\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.248706 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3d01532-a5ff-46fb-b8d2-dd8117802655-ca-trust-extracted\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.248706 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-hub\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7e47df4f-c390-4f91-960b-45f793be4e91-klusterlet-config\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgfl\" (UniqueName: \"kubernetes.io/projected/cd1af325-26fe-4a63-8de9-f7ebe2f189a9-kube-api-access-rtgfl\") pod \"ingress-canary-kmzm2\" (UID: \"cd1af325-26fe-4a63-8de9-f7ebe2f189a9\") " pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.248996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1af325-26fe-4a63-8de9-f7ebe2f189a9-cert\") pod \"ingress-canary-kmzm2\" (UID: \"cd1af325-26fe-4a63-8de9-f7ebe2f189a9\") " pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3d180c6-053f-43a8-8d83-f8f714ea5907-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6\" (UID: \"a3d180c6-053f-43a8-8d83-f8f714ea5907\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4dp\" (UniqueName: \"kubernetes.io/projected/7e47df4f-c390-4f91-960b-45f793be4e91-kube-api-access-8t4dp\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-tmp-dir\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-installation-pull-secrets\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4jr\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-kube-api-access-gw4jr\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249139    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3d01532-a5ff-46fb-b8d2-dd8117802655-ca-trust-extracted\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.249179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f697167f-d912-464a-a29b-48cdd79a9154-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.250066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.249902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f697167f-d912-464a-a29b-48cdd79a9154-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.250436 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.250369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-tmp-dir\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.253149 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253115    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:15.253415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-config-volume\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.253498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-bound-sa-token\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.253498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-ca\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.253597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-image-registry-private-configuration\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.253597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-trusted-ca\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.253597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2k2p\" (UniqueName: \"kubernetes.io/projected/f697167f-d912-464a-a29b-48cdd79a9154-kube-api-access-n2k2p\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.253597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253587    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.253789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-certificates\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.253789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e47df4f-c390-4f91-960b-45f793be4e91-tmp\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.253789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tk5\" (UniqueName: \"kubernetes.io/projected/a3d180c6-053f-43a8-8d83-f8f714ea5907-kube-api-access-86tk5\") pod \"managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6\" (UID: \"a3d180c6-053f-43a8-8d83-f8f714ea5907\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.253789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.253789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.253732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grlsz\" (UniqueName: \"kubernetes.io/projected/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-kube-api-access-grlsz\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.254221 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-hub\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.254322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1af325-26fe-4a63-8de9-f7ebe2f189a9-cert\") pod \"ingress-canary-kmzm2\" (UID: \"cd1af325-26fe-4a63-8de9-f7ebe2f189a9\") " pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.254322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.254444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.254528 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a3d180c6-053f-43a8-8d83-f8f714ea5907-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6\" (UID: \"a3d180c6-053f-43a8-8d83-f8f714ea5907\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.254732 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.254709    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:15.254732 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.254730    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:15.254851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-config-volume\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.254851 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.254785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:15.754766977 +0000 UTC m=+35.655907414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:15.254851 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.254788    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:15.254851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.254794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e47df4f-c390-4f91-960b-45f793be4e91-tmp\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.254851 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.254848    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:17:15.754831769 +0000 UTC m=+35.655972205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:15.255657 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.255136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-trusted-ca\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.255657 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.255239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-certificates\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.255657 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.255418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-installation-pull-secrets\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.255657 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.255614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7e47df4f-c390-4f91-960b-45f793be4e91-klusterlet-config\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.256532 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.256505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f697167f-d912-464a-a29b-48cdd79a9154-ca\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.257022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.257001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-image-registry-private-configuration\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.257577 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.257559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgfl\" (UniqueName: \"kubernetes.io/projected/cd1af325-26fe-4a63-8de9-f7ebe2f189a9-kube-api-access-rtgfl\") pod \"ingress-canary-kmzm2\" (UID: \"cd1af325-26fe-4a63-8de9-f7ebe2f189a9\") " pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.257735 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.257719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4dp\" (UniqueName: \"kubernetes.io/projected/7e47df4f-c390-4f91-960b-45f793be4e91-kube-api-access-8t4dp\") pod \"klusterlet-addon-workmgr-67789c8d86-shws6\" (UID: \"7e47df4f-c390-4f91-960b-45f793be4e91\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.261044 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.261027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4jr\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-kube-api-access-gw4jr\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.265409 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.265384    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tk5\" (UniqueName: \"kubernetes.io/projected/a3d180c6-053f-43a8-8d83-f8f714ea5907-kube-api-access-86tk5\") pod \"managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6\" (UID: \"a3d180c6-053f-43a8-8d83-f8f714ea5907\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.265746 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.265725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2k2p\" (UniqueName: \"kubernetes.io/projected/f697167f-d912-464a-a29b-48cdd79a9154-kube-api-access-n2k2p\") pod \"cluster-proxy-proxy-agent-5c995f78fc-b5r6q\" (UID: \"f697167f-d912-464a-a29b-48cdd79a9154\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.266211 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.266186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grlsz\" (UniqueName: \"kubernetes.io/projected/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-kube-api-access-grlsz\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.266678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.266659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-bound-sa-token\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.347476 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.347403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:15.349302 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.349267    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"
Jul 27 16:17:15.356606 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.356589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:17:15.369696 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.369503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:15.387184 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.387159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmzm2"
Jul 27 16:17:15.541063 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.541032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"]
Jul 27 16:17:15.544944 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.544923    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"]
Jul 27 16:17:15.546574 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:17:15.546495    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf697167f_d912_464a_a29b_48cdd79a9154.slice/crio-789487dc45b5cc5bd0e0265624c86dcc74b3284dd6afcfd3bca8fc32282c38f3 WatchSource:0}: Error finding container 789487dc45b5cc5bd0e0265624c86dcc74b3284dd6afcfd3bca8fc32282c38f3: Status 404 returned error can't find the container with id 789487dc45b5cc5bd0e0265624c86dcc74b3284dd6afcfd3bca8fc32282c38f3
Jul 27 16:17:15.548840 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:17:15.548635    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e47df4f_c390_4f91_960b_45f793be4e91.slice/crio-9627fc36756a7c6259adf161911982a19b2b5f5f4fd86b454f35c65f2d964d62 WatchSource:0}: Error finding container 9627fc36756a7c6259adf161911982a19b2b5f5f4fd86b454f35c65f2d964d62: Status 404 returned error can't find the container with id 9627fc36756a7c6259adf161911982a19b2b5f5f4fd86b454f35c65f2d964d62
Jul 27 16:17:15.549135 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.549116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6"]
Jul 27 16:17:15.551827 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:17:15.551805    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d180c6_053f_43a8_8d83_f8f714ea5907.slice/crio-10e78ca41af827bf739eb15791426696cb4bc871335e0dae4a4ef6aa9ffe00d1 WatchSource:0}: Error finding container 10e78ca41af827bf739eb15791426696cb4bc871335e0dae4a4ef6aa9ffe00d1: Status 404 returned error can't find the container with id 10e78ca41af827bf739eb15791426696cb4bc871335e0dae4a4ef6aa9ffe00d1
Jul 27 16:17:15.556254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.556229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmzm2"]
Jul 27 16:17:15.558152 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:17:15.558130    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1af325_26fe_4a63_8de9_f7ebe2f189a9.slice/crio-ecb18f777cb51a1cde7bed52b4f37ce251a812d31a7dc4e1fded4beae9105a4e WatchSource:0}: Error finding container ecb18f777cb51a1cde7bed52b4f37ce251a812d31a7dc4e1fded4beae9105a4e: Status 404 returned error can't find the container with id ecb18f777cb51a1cde7bed52b4f37ce251a812d31a7dc4e1fded4beae9105a4e
Jul 27 16:17:15.730506 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.730476    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:15.734260 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.734233    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fft22\""
Jul 27 16:17:15.734396 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.734309    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 16:17:15.734667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.734649    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 16:17:15.757964 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.757945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:15.758063 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.757977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:15.758129 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.758099    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:15.758169 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.758148    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:17:16.758134824 +0000 UTC m=+36.659275249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:15.758169 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.758102    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:15.758169 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.758167    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:15.758267 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:15.758194    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:16.758186518 +0000 UTC m=+36.659326946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:15.947006 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.946964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" event={"ID":"f697167f-d912-464a-a29b-48cdd79a9154","Type":"ContainerStarted","Data":"789487dc45b5cc5bd0e0265624c86dcc74b3284dd6afcfd3bca8fc32282c38f3"}
Jul 27 16:17:15.947940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.947909    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmzm2" event={"ID":"cd1af325-26fe-4a63-8de9-f7ebe2f189a9","Type":"ContainerStarted","Data":"ecb18f777cb51a1cde7bed52b4f37ce251a812d31a7dc4e1fded4beae9105a4e"}
Jul 27 16:17:15.948923 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.948901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6" event={"ID":"7e47df4f-c390-4f91-960b-45f793be4e91","Type":"ContainerStarted","Data":"9627fc36756a7c6259adf161911982a19b2b5f5f4fd86b454f35c65f2d964d62"}
Jul 27 16:17:15.949838 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:15.949819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6" event={"ID":"a3d180c6-053f-43a8-8d83-f8f714ea5907","Type":"ContainerStarted","Data":"10e78ca41af827bf739eb15791426696cb4bc871335e0dae4a4ef6aa9ffe00d1"}
Jul 27 16:17:16.734216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.731689    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:16.734216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.731876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:16.735718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.735694    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 16:17:16.735945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.735928    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9d9gv\""
Jul 27 16:17:16.736833 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.736815    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 16:17:16.766582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.766556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:16.766737 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.766684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:16.766863 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:16.766845    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:16.766967 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:16.766918    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:17:18.76689759 +0000 UTC m=+38.668038029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:16.767363 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:16.767345    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:16.767363 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:16.767364    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:16.767538 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:16.767416    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:18.767400602 +0000 UTC m=+38.668541054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:16.904016 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:16.903980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:18.792056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:18.792110    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:18.792224    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:18.792229    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:18.792252    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:18.792309    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:17:22.792274492 +0000 UTC m=+42.693414937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:18.792472 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:18.792331    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:22.792320921 +0000 UTC m=+42.693461354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:20.639639 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:20.639597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:20.742332 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:20.742301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:20.865310 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:20.865262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:22.825047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.824959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:22.825047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.825002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:22.825490 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:22.825131    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:22.825490 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:22.825135    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:22.825490 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:22.825157    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:22.825490 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:22.825191    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:17:30.825171899 +0000 UTC m=+50.726312328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:22.825490 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:22.825214    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:30.825196946 +0000 UTC m=+50.726337374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:22.971440 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.971401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6" event={"ID":"7e47df4f-c390-4f91-960b-45f793be4e91","Type":"ContainerStarted","Data":"4fa8d69962a91997ac8f2e7928b66bc0835ee0b39daa6fd7ec8cb86aff9972d2"}
Jul 27 16:17:22.971682 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.971655    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:22.973777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.973749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6" event={"ID":"a3d180c6-053f-43a8-8d83-f8f714ea5907","Type":"ContainerStarted","Data":"fe665f18a2b1c13022165650754dc0c8df1f4ccbdacbad9fa7184269a8217afd"}
Jul 27 16:17:22.973887 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.973817    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6"
Jul 27 16:17:22.976133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.976113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" event={"ID":"f697167f-d912-464a-a29b-48cdd79a9154","Type":"ContainerStarted","Data":"fce2b0e5c3555e57eff8c8510b6814a70b2ad189e22a3ea5fd8ed6a6348feee1"}
Jul 27 16:17:22.976227 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.976147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" event={"ID":"f697167f-d912-464a-a29b-48cdd79a9154","Type":"ContainerStarted","Data":"f54cc37e2fc4659f2e4907020eecfe598a511720b082e37e91a8d2616a6def43"}
Jul 27 16:17:22.976227 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.976164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" event={"ID":"f697167f-d912-464a-a29b-48cdd79a9154","Type":"ContainerStarted","Data":"cf6d17e5efc838faa62947d0eb2f9b9a2be80c51e78d4318ebbf9a20bcffb188"}
Jul 27 16:17:22.991057 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:22.991007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67789c8d86-shws6" podStartSLOduration=28.21166997 podStartE2EDuration="34.990990528s" totalImagesPullingTime="6.779320558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:22.990899876 +0000 UTC m=+42.892040325" watchObservedRunningTime="2026-07-27 16:17:22.990990528 +0000 UTC m=+42.892130974"
Jul 27 16:17:23.015340 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:23.015268    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" podStartSLOduration=28.256747644 podStartE2EDuration="35.015247552s" totalImagesPullingTime="6.758499908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:23.014090105 +0000 UTC m=+42.915230554" watchObservedRunningTime="2026-07-27 16:17:23.015247552 +0000 UTC m=+42.916388001"
Jul 27 16:17:23.031492 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:23.031450    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7fcff6cf86-tl7m6" podStartSLOduration=28.277711716 podStartE2EDuration="35.031435732s" totalImagesPullingTime="6.753724016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:23.030987568 +0000 UTC m=+42.932128020" watchObservedRunningTime="2026-07-27 16:17:23.031435732 +0000 UTC m=+42.932576174"
Jul 27 16:17:24.983028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:24.982991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmzm2" event={"ID":"cd1af325-26fe-4a63-8de9-f7ebe2f189a9","Type":"ContainerStarted","Data":"616ca25702495f87a195e7c3f8d09bbd9c56d8b50628c29946e6e25570dfc078"}
Jul 27 16:17:25.015904 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:25.015810    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kmzm2" podStartSLOduration=0.812529278 podStartE2EDuration="10.015791707s" totalImagesPullingTime="9.203262429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:25.014216749 +0000 UTC m=+44.915357197" watchObservedRunningTime="2026-07-27 16:17:25.015791707 +0000 UTC m=+44.916932155"
Jul 27 16:17:25.230742 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:25.230710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:25.330346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:25.330255    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:25.420581 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:25.420543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:26.231146 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:26.231093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:26.310467 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:26.310402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:28.194273 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:28.194235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:28.293842 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:28.293792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:28.396110 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:28.396072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:29.125579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:29.125550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:29.225730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:29.225695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:30.887257 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:30.887222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:30.887775 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:30.887355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:30.887775 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:30.887385    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:30.887775 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:30.887472    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:17:46.887454165 +0000 UTC m=+66.788594591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:30.887775 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:30.887487    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:30.887775 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:30.887502    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:30.887775 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:30.887555    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:46.887537438 +0000 UTC m=+66.788677885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:34.248346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:34.248294    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:37.179840 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:37.179803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:37.266898 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:37.266860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:37.399579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:37.399541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:37.501344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:37.501245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:40.946172 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:40.946144    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nc8mh"
Jul 27 16:17:41.019503 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:41.019468    2559 generic.go:356] "Generic (PLEG): container finished" podID="e00e9260-9cdf-4b60-9185-121d37a15afa" containerID="80b06ed5598d51832a8406a0ca523b5f135064661262183edaaaae365139b642" exitCode=0
Jul 27 16:17:41.019659 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:41.019509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerDied","Data":"80b06ed5598d51832a8406a0ca523b5f135064661262183edaaaae365139b642"}
Jul 27 16:17:41.128215 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:41.128175    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:41.215361 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:41.215241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:41.352592 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:41.352556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:41.413474 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:41.413439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:42.023985 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:42.023906    2559 generic.go:356] "Generic (PLEG): container finished" podID="e00e9260-9cdf-4b60-9185-121d37a15afa" containerID="80b4e23966b1a524ab485f0c53d26f59e962ef70a5e473c9110ef2c53b7ac6b7" exitCode=0
Jul 27 16:17:42.023985 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:42.023971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerDied","Data":"80b4e23966b1a524ab485f0c53d26f59e962ef70a5e473c9110ef2c53b7ac6b7"}
Jul 27 16:17:42.296038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:42.295953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:42.396001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:42.395956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:43.028687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:43.028651    2559 generic.go:356] "Generic (PLEG): container finished" podID="e00e9260-9cdf-4b60-9185-121d37a15afa" containerID="41b00de508d254b3fe156546218834bacf648f04118f77a0058dfc78282fc2ad" exitCode=0
Jul 27 16:17:43.029053 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:43.028730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerDied","Data":"41b00de508d254b3fe156546218834bacf648f04118f77a0058dfc78282fc2ad"}
Jul 27 16:17:45.934880 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:45.934840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:46.035305 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.035228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:46.313447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.313363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:46.317305 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.316828    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 16:17:46.326344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.326320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3357411f-3fd8-4a76-9d8d-e5af12e69251-original-pull-secret\") pod \"global-pull-secret-syncer-bbqvv\" (UID: \"3357411f-3fd8-4a76-9d8d-e5af12e69251\") " pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:46.414885 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.414842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:17:46.418058 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.418033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 16:17:46.425519 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.425498    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 16:17:46.425618 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.425574    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs podName:6f2d7be9-b333-4690-97ec-820cc701173b nodeName:}" failed. No retries permitted until 2026-07-27 16:18:50.425553984 +0000 UTC m=+130.326694430 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs") pod "network-metrics-daemon-nw2nf" (UID: "6f2d7be9-b333-4690-97ec-820cc701173b") : secret "metrics-daemon-secret" not found
Jul 27 16:17:46.450022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.449991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bbqvv"
Jul 27 16:17:46.515435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.515400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:46.518875 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.518851    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 16:17:46.529457 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.529432    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 16:17:46.539601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.539578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzj5\" (UniqueName: \"kubernetes.io/projected/d55354b4-55ba-4fed-bab5-7035641c759e-kube-api-access-dgzj5\") pod \"network-check-target-hblhv\" (UID: \"d55354b4-55ba-4fed-bab5-7035641c759e\") " pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:46.643062 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.643024    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fft22\""
Jul 27 16:17:46.650429 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.650402    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.918567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.918658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.918890    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.918956    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:18:18.91893722 +0000 UTC m=+98.820077663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.918988    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.919002    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:17:46.919469 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:17:46.919043    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:18.919027693 +0000 UTC m=+98.820168124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:17:46.943262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.943203    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bbqvv"]
Jul 27 16:17:46.948063 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:17:46.948026    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3357411f_3fd8_4a76_9d8d_e5af12e69251.slice/crio-cd07070c95b49b78e328e82ba24b65f8185f3c81d1171dc1d964d6de61c71f1d WatchSource:0}: Error finding container cd07070c95b49b78e328e82ba24b65f8185f3c81d1171dc1d964d6de61c71f1d: Status 404 returned error can't find the container with id cd07070c95b49b78e328e82ba24b65f8185f3c81d1171dc1d964d6de61c71f1d
Jul 27 16:17:46.972732 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:46.972702    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hblhv"]
Jul 27 16:17:46.976807 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:17:46.976777    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55354b4_55ba_4fed_bab5_7035641c759e.slice/crio-3f3ceaff1410856070cbdb3539d204518c9f1bc7b0ee22f01410727c6cdb5ee2 WatchSource:0}: Error finding container 3f3ceaff1410856070cbdb3539d204518c9f1bc7b0ee22f01410727c6cdb5ee2: Status 404 returned error can't find the container with id 3f3ceaff1410856070cbdb3539d204518c9f1bc7b0ee22f01410727c6cdb5ee2
Jul 27 16:17:47.040441 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:47.040407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbqvv" event={"ID":"3357411f-3fd8-4a76-9d8d-e5af12e69251","Type":"ContainerStarted","Data":"cd07070c95b49b78e328e82ba24b65f8185f3c81d1171dc1d964d6de61c71f1d"}
Jul 27 16:17:47.043651 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:47.043621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerStarted","Data":"dd7956bdf475c2ef61831f1495b4d6252dc1ffd82da91b9a706d83f91cfb074a"}
Jul 27 16:17:47.044868 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:47.044845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hblhv" event={"ID":"d55354b4-55ba-4fed-bab5-7035641c759e","Type":"ContainerStarted","Data":"3f3ceaff1410856070cbdb3539d204518c9f1bc7b0ee22f01410727c6cdb5ee2"}
Jul 27 16:17:48.051178 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:48.051138    2559 generic.go:356] "Generic (PLEG): container finished" podID="e00e9260-9cdf-4b60-9185-121d37a15afa" containerID="dd7956bdf475c2ef61831f1495b4d6252dc1ffd82da91b9a706d83f91cfb074a" exitCode=0
Jul 27 16:17:48.051667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:48.051236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerDied","Data":"dd7956bdf475c2ef61831f1495b4d6252dc1ffd82da91b9a706d83f91cfb074a"}
Jul 27 16:17:49.057484 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:49.057452    2559 generic.go:356] "Generic (PLEG): container finished" podID="e00e9260-9cdf-4b60-9185-121d37a15afa" containerID="2607af6431dbd9bc30957e2baa75d0971d8ff1c1f1b2f72097a561c5cafded3a" exitCode=0
Jul 27 16:17:49.057901 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:49.057512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerDied","Data":"2607af6431dbd9bc30957e2baa75d0971d8ff1c1f1b2f72097a561c5cafded3a"}
Jul 27 16:17:50.067343 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:50.067312    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" event={"ID":"e00e9260-9cdf-4b60-9185-121d37a15afa","Type":"ContainerStarted","Data":"485094dd0d52a9c594c2ed10b980dd270f3ce24578995c4c3cda270af5132fde"}
Jul 27 16:17:51.244833 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:51.244788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:51.308032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:51.307980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:17:52.077771 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:52.077727    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bbqvv" event={"ID":"3357411f-3fd8-4a76-9d8d-e5af12e69251","Type":"ContainerStarted","Data":"73d8f0af9cee26f392289efe157ce27f49110f1056178d1cb2bf22e4e1563bf4"}
Jul 27 16:17:52.079261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:52.079231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hblhv" event={"ID":"d55354b4-55ba-4fed-bab5-7035641c759e","Type":"ContainerStarted","Data":"bd9a9b62923de502db1dc2812fa181e742c2d2e246f2a5d7e86c707046ee6028"}
Jul 27 16:17:52.079404 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:52.079380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:17:52.095963 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:52.095912    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qfcd7" podStartSLOduration=9.583334252 podStartE2EDuration="1m11.095896695s" totalImagesPullingTime="1m1.512562443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:51.1081512 +0000 UTC m=+71.009291658" watchObservedRunningTime="2026-07-27 16:17:52.095896695 +0000 UTC m=+71.997037193"
Jul 27 16:17:52.096428 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:52.096398    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bbqvv" podStartSLOduration=66.141467149 podStartE2EDuration="1m11.096390399s" totalImagesPullingTime="4.95492325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:52.096031984 +0000 UTC m=+71.997172432" watchObservedRunningTime="2026-07-27 16:17:52.096390399 +0000 UTC m=+71.997530848"
Jul 27 16:17:52.115615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:52.115565    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hblhv" podStartSLOduration=66.427289672 podStartE2EDuration="1m11.115548407s" totalImagesPullingTime="4.688258735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:52.114941281 +0000 UTC m=+72.016081729" watchObservedRunningTime="2026-07-27 16:17:52.115548407 +0000 UTC m=+72.016688856"
Jul 27 16:17:59.427354 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:17:59.427313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:02.271180 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:02.271141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:02.538156 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:02.538075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:08.195543 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:08.195504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:08.250553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:08.250516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:08.314543 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:08.314510    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:18.958760 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:18.958726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:18:18.959096 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:18.958795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") pod \"image-registry-5c69f7688b-52vqs\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") " pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:18:18.959096 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:18.958875    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:18:18.959096 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:18.958950    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls podName:419fc3b7-5ea7-46c6-8b4e-cc2a82958adc nodeName:}" failed. No retries permitted until 2026-07-27 16:19:22.9589334 +0000 UTC m=+162.860073826 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls") pod "dns-default-pcfdm" (UID: "419fc3b7-5ea7-46c6-8b4e-cc2a82958adc") : secret "dns-default-metrics-tls" not found
Jul 27 16:18:18.959096 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:18.958881    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:18:18.959096 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:18.958975    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c69f7688b-52vqs: secret "image-registry-tls" not found
Jul 27 16:18:18.959096 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:18.959024    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls podName:c3d01532-a5ff-46fb-b8d2-dd8117802655 nodeName:}" failed. No retries permitted until 2026-07-27 16:19:22.959013962 +0000 UTC m=+162.860154388 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls") pod "image-registry-5c69f7688b-52vqs" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655") : secret "image-registry-tls" not found
Jul 27 16:18:21.231118 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.231084    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"]
Jul 27 16:18:21.260107 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.260077    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"]
Jul 27 16:18:21.260238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.260165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"
Jul 27 16:18:21.263337 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.263319    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qtkrl\""
Jul 27 16:18:21.376053 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.376014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmqs\" (UniqueName: \"kubernetes.io/projected/7347a713-2fc7-4210-ae64-1b3da09936df-kube-api-access-zkmqs\") pod \"network-check-source-7d88cb55d8-dkwmb\" (UID: \"7347a713-2fc7-4210-ae64-1b3da09936df\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"
Jul 27 16:18:21.476807 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.476771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmqs\" (UniqueName: \"kubernetes.io/projected/7347a713-2fc7-4210-ae64-1b3da09936df-kube-api-access-zkmqs\") pod \"network-check-source-7d88cb55d8-dkwmb\" (UID: \"7347a713-2fc7-4210-ae64-1b3da09936df\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"
Jul 27 16:18:21.487119 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.487051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmqs\" (UniqueName: \"kubernetes.io/projected/7347a713-2fc7-4210-ae64-1b3da09936df-kube-api-access-zkmqs\") pod \"network-check-source-7d88cb55d8-dkwmb\" (UID: \"7347a713-2fc7-4210-ae64-1b3da09936df\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"
Jul 27 16:18:21.515535 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.515502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:21.569394 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.569368    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"
Jul 27 16:18:21.618084 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.618049    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:21.710379 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:21.710349    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb"]
Jul 27 16:18:21.713303 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:18:21.713263    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7347a713_2fc7_4210_ae64_1b3da09936df.slice/crio-f26b26598ceb09ad39d98b60b69b3cdf0a679e7de109dd1b79df142903fe638e WatchSource:0}: Error finding container f26b26598ceb09ad39d98b60b69b3cdf0a679e7de109dd1b79df142903fe638e: Status 404 returned error can't find the container with id f26b26598ceb09ad39d98b60b69b3cdf0a679e7de109dd1b79df142903fe638e
Jul 27 16:18:22.174872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:22.174837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb" event={"ID":"7347a713-2fc7-4210-ae64-1b3da09936df","Type":"ContainerStarted","Data":"083605ec3474aa1ab8fe9cf094e725123c8b4056c5e7a1c35b456b99841b82b8"}
Jul 27 16:18:22.175051 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:22.174887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb" event={"ID":"7347a713-2fc7-4210-ae64-1b3da09936df","Type":"ContainerStarted","Data":"f26b26598ceb09ad39d98b60b69b3cdf0a679e7de109dd1b79df142903fe638e"}
Jul 27 16:18:22.194874 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:22.194828    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dkwmb" podStartSLOduration=1.194813938 podStartE2EDuration="1.194813938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:22.193549836 +0000 UTC m=+102.094690284" watchObservedRunningTime="2026-07-27 16:18:22.194813938 +0000 UTC m=+102.095954386"
Jul 27 16:18:23.088800 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:23.088771    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hblhv"
Jul 27 16:18:24.362093 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:24.362037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:24.462702 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:24.462674    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:25.331412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:25.331375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:25.430384 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:25.430344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:25.530964 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:25.530919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:25.752354 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:25.752312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:25.810184 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:25.810147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:26.029205 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.029114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:26.129264 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.129232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:26.478600 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.478560    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:26.559261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.559225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:26.754723 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.754651    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z2m9g"]
Jul 27 16:18:26.757389 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.757373    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:26.761207 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.761187    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 16:18:26.761958 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.761942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 16:18:26.762041 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.761944    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 16:18:26.762041 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.761944    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 16:18:26.763212 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.763187    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m7sx7\""
Jul 27 16:18:26.788805 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.788780    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z2m9g"]
Jul 27 16:18:26.801122 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.801092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:26.915104 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.915071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0eb2388-6a44-436c-ae6c-bf13dce0856e-crio-socket\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:26.915104 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.915104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzhw\" (UniqueName: \"kubernetes.io/projected/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-api-access-8zzhw\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:26.915377 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.915154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:26.915377 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.915251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:26.915377 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.915311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0eb2388-6a44-436c-ae6c-bf13dce0856e-data-volume\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:26.920226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:26.920199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:27.016009 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.015936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0eb2388-6a44-436c-ae6c-bf13dce0856e-data-volume\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016009 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.015992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0eb2388-6a44-436c-ae6c-bf13dce0856e-crio-socket\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.016023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzhw\" (UniqueName: \"kubernetes.io/projected/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-api-access-8zzhw\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.016085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.016125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.016136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0eb2388-6a44-436c-ae6c-bf13dce0856e-crio-socket\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016414 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.016234    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:27.016414 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.016304    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:27.016414 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.016321    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0eb2388-6a44-436c-ae6c-bf13dce0856e-data-volume\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.016414 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.016328    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:27.516308761 +0000 UTC m=+107.417449188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:27.016414 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.016378    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:27.516359829 +0000 UTC m=+107.417500259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:27.025931 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.025911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzhw\" (UniqueName: \"kubernetes.io/projected/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-api-access-8zzhw\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.519759 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.519723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.520154 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.519775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:27.520154 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.519856    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:27.520154 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.519869    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:27.520154 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.519911    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.519896892 +0000 UTC m=+108.421037317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:27.520154 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:27.519935    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.519922041 +0000 UTC m=+108.421062466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:27.695143 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:27.695104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:28.181664 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:28.181627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:28.529581 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:28.529492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:28.529581 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:28.529548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:28.530011 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:28.529620    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:28.530011 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:28.529693    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:30.529676034 +0000 UTC m=+110.430816460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:28.530011 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:28.529621    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:28.530011 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:28.529786    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:30.529768535 +0000 UTC m=+110.430908962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:29.251633 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:29.251593    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:29.493050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:29.493013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:29.712635 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:29.712609    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2k8f_bf283595-3a83-4674-af35-b7861330a709/dns-node-resolver/0.log"
Jul 27 16:18:29.730968 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:29.730938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:29.840276 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:29.840242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:29.912007 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:29.911979    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:30.505906 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:30.505881    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmfdr_f795eac2-9fea-403e-ba0e-54c90483cc72/node-ca/0.log"
Jul 27 16:18:30.545228 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:30.545201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:30.545342 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:30.545316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:30.545342 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:30.545323    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:30.545415 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:30.545388    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:34.545373361 +0000 UTC m=+114.446513787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:30.545452 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:30.545435    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:30.545505 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:30.545494    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:34.545482505 +0000 UTC m=+114.446622931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:30.794190 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:30.794106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:31.210061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:31.210023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:31.305680 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:31.305652    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmzm2_cd1af325-26fe-4a63-8de9-f7ebe2f189a9/serve-healthcheck-canary/0.log"
Jul 27 16:18:31.438540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:31.438503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:31.590494 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:31.590401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:31.997772 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:31.997732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:32.610197 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:32.610138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:33.197769 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:33.197727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:33.302118 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:33.302088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:33.401338 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:33.401305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:33.478660 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:33.478577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:33.796014 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:33.795943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:33.992826 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:33.992785    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:34.394940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:34.394901    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:34.503954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:34.503922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:34.574758 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:34.574727    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:34.574950 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:34.574769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:34.574950 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:34.574897    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:34.574950 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:34.574916    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:34.575108 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:34.574972    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:42.574952356 +0000 UTC m=+122.476092808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:34.575108 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:18:34.574992    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm podName:a0eb2388-6a44-436c-ae6c-bf13dce0856e nodeName:}" failed. No retries permitted until 2026-07-27 16:18:42.574982542 +0000 UTC m=+122.476122974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z2m9g" (UID: "a0eb2388-6a44-436c-ae6c-bf13dce0856e") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:34.987549 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:34.987513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:35.415487 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:35.415453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:35.783107 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:35.783014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:36.385563 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:36.385532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:36.455869 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:36.455834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:36.812056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:36.812024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:37.615692 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:37.615661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:37.819185 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:37.819147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:38.011490 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:38.011456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:38.456046 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:38.456007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:38.589675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:38.589641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:38.691202 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:38.691169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:39.425115 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:39.425074    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:39.802187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:39.802086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:40.048062 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:40.048024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:40.201954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:40.201925    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:42.635814 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.635775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:42.636190 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.635827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:42.636305 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.636267    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0eb2388-6a44-436c-ae6c-bf13dce0856e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:42.638036 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.638010    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0eb2388-6a44-436c-ae6c-bf13dce0856e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z2m9g\" (UID: \"a0eb2388-6a44-436c-ae6c-bf13dce0856e\") " pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:42.669181 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.669154    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m7sx7\""
Jul 27 16:18:42.676272 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.676252    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z2m9g"
Jul 27 16:18:42.810804 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:42.810771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z2m9g"]
Jul 27 16:18:42.814072 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:18:42.814049    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0eb2388_6a44_436c_ae6c_bf13dce0856e.slice/crio-5927de389be8cdc221089d9f68b5eaeb82f0e899d8f1ed31a955895171a945e9 WatchSource:0}: Error finding container 5927de389be8cdc221089d9f68b5eaeb82f0e899d8f1ed31a955895171a945e9: Status 404 returned error can't find the container with id 5927de389be8cdc221089d9f68b5eaeb82f0e899d8f1ed31a955895171a945e9
Jul 27 16:18:43.236478 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:43.236441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2m9g" event={"ID":"a0eb2388-6a44-436c-ae6c-bf13dce0856e","Type":"ContainerStarted","Data":"5927de389be8cdc221089d9f68b5eaeb82f0e899d8f1ed31a955895171a945e9"}
Jul 27 16:18:43.890609 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:43.890573    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:44.011037 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:44.011001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:44.134435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:44.134395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:44.321893 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:44.321797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:48.253834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:48.253801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2m9g" event={"ID":"a0eb2388-6a44-436c-ae6c-bf13dce0856e","Type":"ContainerStarted","Data":"0a9e389375aebc4d69b6127a7c5675af86ec618a2e69691dbfecc4fdb8e764b2"}
Jul 27 16:18:50.119447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:50.119408    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:50.498256 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:50.498217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:18:50.500522 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:50.500502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d7be9-b333-4690-97ec-820cc701173b-metrics-certs\") pod \"network-metrics-daemon-nw2nf\" (UID: \"6f2d7be9-b333-4690-97ec-820cc701173b\") " pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:18:50.661736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:50.661707    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9d9gv\""
Jul 27 16:18:50.669699 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:50.669673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nw2nf"
Jul 27 16:18:50.816460 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:50.816374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nw2nf"]
Jul 27 16:18:50.820309 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:18:50.820271    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2d7be9_b333_4690_97ec_820cc701173b.slice/crio-0db96f525e26c6ca87c5d3adb0b4c626f526cd8f959f95d3040f2b4e4dd0535d WatchSource:0}: Error finding container 0db96f525e26c6ca87c5d3adb0b4c626f526cd8f959f95d3040f2b4e4dd0535d: Status 404 returned error can't find the container with id 0db96f525e26c6ca87c5d3adb0b4c626f526cd8f959f95d3040f2b4e4dd0535d
Jul 27 16:18:51.268050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:51.265330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nw2nf" event={"ID":"6f2d7be9-b333-4690-97ec-820cc701173b","Type":"ContainerStarted","Data":"0db96f525e26c6ca87c5d3adb0b4c626f526cd8f959f95d3040f2b4e4dd0535d"}
Jul 27 16:18:51.403264 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:51.403224    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:53.276951 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:53.276906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nw2nf" event={"ID":"6f2d7be9-b333-4690-97ec-820cc701173b","Type":"ContainerStarted","Data":"83b93f8352e4169181cd981baa345295091bcf20b86f51e25f6b15c61949882c"}
Jul 27 16:18:53.277401 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:53.276985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nw2nf" event={"ID":"6f2d7be9-b333-4690-97ec-820cc701173b","Type":"ContainerStarted","Data":"e9a98ee60eab9d5e2a718c7769804fe35372efe071dcf4f3e2b5a597ef60017c"}
Jul 27 16:18:53.528705 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:53.528625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:53.628398 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:53.628335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:54.816549 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:54.816509    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:54.928131 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:54.928084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:18:55.285671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:55.285629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2m9g" event={"ID":"a0eb2388-6a44-436c-ae6c-bf13dce0856e","Type":"ContainerStarted","Data":"ef6358cbcd501c1d2ef4af2b75113706d37ea823bc7f0685472ae58fedf64803"}
Jul 27 16:18:55.358368 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:55.358326    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" podUID="f697167f-d912-464a-a29b-48cdd79a9154" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 16:18:58.296861 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:58.296826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z2m9g" event={"ID":"a0eb2388-6a44-436c-ae6c-bf13dce0856e","Type":"ContainerStarted","Data":"d334fd57f879902b203fc41ff79fe4a99699f5e77916200dd1fdeee6d0927c29"}
Jul 27 16:18:58.322521 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:58.322473    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nw2nf" podStartSLOduration=135.975470844 podStartE2EDuration="2m17.322453057s" totalImagesPullingTime="1.346982213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:53.314596317 +0000 UTC m=+133.215736765" watchObservedRunningTime="2026-07-27 16:18:58.322453057 +0000 UTC m=+138.223593506"
Jul 27 16:18:58.323091 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:18:58.323055    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z2m9g" podStartSLOduration=17.633639561 podStartE2EDuration="32.323042449s" totalImagesPullingTime="14.689402888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:18:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:58.322446276 +0000 UTC m=+138.223586729" watchObservedRunningTime="2026-07-27 16:18:58.323042449 +0000 UTC m=+138.224182897"
Jul 27 16:19:00.553393 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:00.553357    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:03.427865 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.427828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:03.553164 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.553130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:03.774860 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.774783    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s55g5"]
Jul 27 16:19:03.776894 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.776878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.779842 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.779821    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 16:19:03.779975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.779856    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 16:19:03.780107 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.780093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 16:19:03.780409 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.780396    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 16:19:03.780493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.780473    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wzslv\""
Jul 27 16:19:03.780816 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.780804    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 16:19:03.781144 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.781131    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 16:19:03.833863 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.833832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:03.904896 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.904862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-root\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.904905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-textfile\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.904930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-wtmp\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.904989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-sys\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.905011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-tls\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.905033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.905058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s8c\" (UniqueName: \"kubernetes.io/projected/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-kube-api-access-27s8c\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.905080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.905158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.905127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-metrics-client-ca\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:03.929783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:03.929740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:04.006367 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006329    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-root\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006367 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006364    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-textfile\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-wtmp\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-root\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-sys\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-tls\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006539    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-wtmp\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006865 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-sys\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006865 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s8c\" (UniqueName: \"kubernetes.io/projected/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-kube-api-access-27s8c\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006865 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006865 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-metrics-client-ca\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.006865 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.006761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-textfile\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.007302 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.007269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.007418 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.007301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-metrics-client-ca\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.009325 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.009305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.009467 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.009448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-node-exporter-tls\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.015538 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.015517    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s8c\" (UniqueName: \"kubernetes.io/projected/3c19f12f-07c3-470e-bb1c-cdf124c2cdf2-kube-api-access-27s8c\") pod \"node-exporter-s55g5\" (UID: \"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2\") " pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.029158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.029063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:04.088943 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.088918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s55g5"
Jul 27 16:19:04.097861 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:19:04.097832    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c19f12f_07c3_470e_bb1c_cdf124c2cdf2.slice/crio-8199fae8d7a728987c7af322455d956a1dd399e512d8324e417d0b4af1854c7d WatchSource:0}: Error finding container 8199fae8d7a728987c7af322455d956a1dd399e512d8324e417d0b4af1854c7d: Status 404 returned error can't find the container with id 8199fae8d7a728987c7af322455d956a1dd399e512d8324e417d0b4af1854c7d
Jul 27 16:19:04.134071 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.134041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:04.316436 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.316357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s55g5" event={"ID":"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2","Type":"ContainerStarted","Data":"8199fae8d7a728987c7af322455d956a1dd399e512d8324e417d0b4af1854c7d"}
Jul 27 16:19:04.563129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.563095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:04.693081 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.693039    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:04.807177 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:04.807141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:05.028241 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:05.028155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:05.321344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:05.321246    2559 generic.go:356] "Generic (PLEG): container finished" podID="3c19f12f-07c3-470e-bb1c-cdf124c2cdf2" containerID="cfc589a068ed283082ecc1e0b31dc557eb55e89c623ecc9c11618e850f18ed0e" exitCode=0
Jul 27 16:19:05.321344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:05.321315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s55g5" event={"ID":"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2","Type":"ContainerDied","Data":"cfc589a068ed283082ecc1e0b31dc557eb55e89c623ecc9c11618e850f18ed0e"}
Jul 27 16:19:05.357923 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:05.357891    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" podUID="f697167f-d912-464a-a29b-48cdd79a9154" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 16:19:06.326712 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.326677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s55g5" event={"ID":"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2","Type":"ContainerStarted","Data":"05d392d8a8b2a33c56aa53c9e9b15a0f57d3426dbad4cf6bb1537392f1b23359"}
Jul 27 16:19:06.327085 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.326729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s55g5" event={"ID":"3c19f12f-07c3-470e-bb1c-cdf124c2cdf2","Type":"ContainerStarted","Data":"3db8017157cfeef858c48fe55d55fbc3db2b923219a50c8d6a20f842b8e9657c"}
Jul 27 16:19:06.357549 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.357501    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s55g5" podStartSLOduration=2.325044304 podStartE2EDuration="3.357484004s" totalImagesPullingTime="970.175445ms" totalInitContainerRuntime="62.264255ms" isStatefulPod=true podCreationTimestamp="2026-07-27 16:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:06.356909218 +0000 UTC m=+146.258049667" watchObservedRunningTime="2026-07-27 16:19:06.357484004 +0000 UTC m=+146.258624451"
Jul 27 16:19:06.674388 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.674358    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:06.758101 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.758067    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:06.827919 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.827880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:06.938764 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:06.938670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:07.140270 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:07.140224    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:07.639967 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:07.639936    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:07.743636 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:07.743600    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:07.838035 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:07.837996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:07.938583 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:07.938549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:08.101240 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:08.101208    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:08.201365 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:08.201268    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:08.302481 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:08.302452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:08.436153 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:08.436112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:08.712058 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:08.712021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:08.810858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:08.810826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:09.119934 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:09.119849    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:09.223409 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:09.220457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:09.320705 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:09.320671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:09.724018 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:09.723977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:09.933749 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:09.933720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:10.014214 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:10.014126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:13.209895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:13.209856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:13.309375 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:13.309331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:13.451631 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:13.451598    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c69f7688b-52vqs"]
Jul 27 16:19:13.451852 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:19:13.451833    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5c69f7688b-52vqs" podUID="c3d01532-a5ff-46fb-b8d2-dd8117802655"
Jul 27 16:19:13.578201 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:13.578123    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:13.676663 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:13.676626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:13.777616 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:13.777581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:14.350180 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.350154    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:19:14.356328 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.356300    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:19:14.494028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.493999    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-installation-pull-secrets\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw4jr\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-kube-api-access-gw4jr\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494053    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-bound-sa-token\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494080    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3d01532-a5ff-46fb-b8d2-dd8117802655-ca-trust-extracted\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494122    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-image-registry-private-configuration\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494156    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-trusted-ca\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494183    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-certificates\") pod \"c3d01532-a5ff-46fb-b8d2-dd8117802655\" (UID: \"c3d01532-a5ff-46fb-b8d2-dd8117802655\") "
Jul 27 16:19:14.494536 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494491    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d01532-a5ff-46fb-b8d2-dd8117802655-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:19:14.494655 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:19:14.494738 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.494667    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:19:14.496500 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.496470    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-kube-api-access-gw4jr" (OuterVolumeSpecName: "kube-api-access-gw4jr") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "kube-api-access-gw4jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:19:14.496619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.496530    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:19:14.496701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.496677    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:19:14.496763 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.496744    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c3d01532-a5ff-46fb-b8d2-dd8117802655" (UID: "c3d01532-a5ff-46fb-b8d2-dd8117802655"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:19:14.595276 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595243    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-installation-pull-secrets\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.595276 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595268    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4jr\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-kube-api-access-gw4jr\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.595276 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595289    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-bound-sa-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.595505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595305    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c3d01532-a5ff-46fb-b8d2-dd8117802655-ca-trust-extracted\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.595505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595319    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c3d01532-a5ff-46fb-b8d2-dd8117802655-image-registry-private-configuration\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.595505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595330    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-trusted-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.595505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.595338    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-certificates\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:14.653497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.653465    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:14.756516 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.756479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:14.888033 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:14.887998    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:15.352793 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.352764    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c69f7688b-52vqs"
Jul 27 16:19:15.358494 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.358464    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" podUID="f697167f-d912-464a-a29b-48cdd79a9154" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 27 16:19:15.358608 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.358524    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q"
Jul 27 16:19:15.358954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.358926    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fce2b0e5c3555e57eff8c8510b6814a70b2ad189e22a3ea5fd8ed6a6348feee1"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 27 16:19:15.358993 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.358974    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" podUID="f697167f-d912-464a-a29b-48cdd79a9154" containerName="service-proxy" containerID="cri-o://fce2b0e5c3555e57eff8c8510b6814a70b2ad189e22a3ea5fd8ed6a6348feee1" gracePeriod=30
Jul 27 16:19:15.405497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.405471    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c69f7688b-52vqs"]
Jul 27 16:19:15.412054 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.412026    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c69f7688b-52vqs"]
Jul 27 16:19:15.502565 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.502423    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c3d01532-a5ff-46fb-b8d2-dd8117802655-registry-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:19:15.989044 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:15.989010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:16.057973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:16.057925    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:16.358735 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:16.358657    2559 generic.go:356] "Generic (PLEG): container finished" podID="f697167f-d912-464a-a29b-48cdd79a9154" containerID="fce2b0e5c3555e57eff8c8510b6814a70b2ad189e22a3ea5fd8ed6a6348feee1" exitCode=2
Jul 27 16:19:16.358735 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:16.358719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" event={"ID":"f697167f-d912-464a-a29b-48cdd79a9154","Type":"ContainerDied","Data":"fce2b0e5c3555e57eff8c8510b6814a70b2ad189e22a3ea5fd8ed6a6348feee1"}
Jul 27 16:19:16.359102 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:16.358765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c995f78fc-b5r6q" event={"ID":"f697167f-d912-464a-a29b-48cdd79a9154","Type":"ContainerStarted","Data":"74c28c8ca6bedbdb5e84f8bef2c78e2530c8524a478417af7a737b4a44647124"}
Jul 27 16:19:16.737494 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:16.737454    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d01532-a5ff-46fb-b8d2-dd8117802655" path="/var/lib/kubelet/pods/c3d01532-a5ff-46fb-b8d2-dd8117802655/volumes"
Jul 27 16:19:17.723733 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:17.723696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:17.828935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:17.828897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:18.082933 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:19:18.082851    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pcfdm" podUID="419fc3b7-5ea7-46c6-8b4e-cc2a82958adc"
Jul 27 16:19:18.366047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:18.365960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:19:20.596714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:20.596672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:20.708592 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:20.708556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:20.809447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:20.809412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:23.058663 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:23.058623    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:19:23.060868 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:23.060845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/419fc3b7-5ea7-46c6-8b4e-cc2a82958adc-metrics-tls\") pod \"dns-default-pcfdm\" (UID: \"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc\") " pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:19:23.169955 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:23.169923    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2mnq\""
Jul 27 16:19:23.177288 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:23.177262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:19:23.321238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:23.321209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcfdm"]
Jul 27 16:19:23.323768 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:19:23.323743    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419fc3b7_5ea7_46c6_8b4e_cc2a82958adc.slice/crio-c1ddeded75b1b3c63bf362b76d24c775531d003499171f6aca5a9032ae185a3a WatchSource:0}: Error finding container c1ddeded75b1b3c63bf362b76d24c775531d003499171f6aca5a9032ae185a3a: Status 404 returned error can't find the container with id c1ddeded75b1b3c63bf362b76d24c775531d003499171f6aca5a9032ae185a3a
Jul 27 16:19:23.381894 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:23.381869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcfdm" event={"ID":"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc","Type":"ContainerStarted","Data":"c1ddeded75b1b3c63bf362b76d24c775531d003499171f6aca5a9032ae185a3a"}
Jul 27 16:19:24.341017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:24.340983    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:24.443935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:24.443896    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:25.393454 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:25.393417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcfdm" event={"ID":"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc","Type":"ContainerStarted","Data":"9c134a7133e07e021cba745ff300ed0f563ae6c3d7ec60ff7dcc77f758e60dd4"}
Jul 27 16:19:25.393820 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:25.393492    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcfdm" event={"ID":"419fc3b7-5ea7-46c6-8b4e-cc2a82958adc","Type":"ContainerStarted","Data":"81b9632b4fa3192ed5004efedc3a71dffee20d6e7e5f70580e718ec55dbd2dba"}
Jul 27 16:19:25.393820 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:25.393536    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:19:25.418047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:25.417999    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pcfdm" podStartSLOduration=128.770089151 podStartE2EDuration="2m10.417985758s" totalImagesPullingTime="1.647896607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:25.417145795 +0000 UTC m=+165.318286243" watchObservedRunningTime="2026-07-27 16:19:25.417985758 +0000 UTC m=+165.319126206"
Jul 27 16:19:27.594381 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:27.594347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:27.651701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:27.651669    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:29.124796 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:29.124761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:29.225330 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:29.225290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:30.319653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.319621    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866488f997-8gl6s"]
Jul 27 16:19:30.334984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.334950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.335141 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.335001    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866488f997-8gl6s"]
Jul 27 16:19:30.337748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.337730    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 16:19:30.337748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.337743    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bqpqf\""
Jul 27 16:19:30.338974 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.338951    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 16:19:30.339085 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.339002    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 16:19:30.339085 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.339010    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 16:19:30.339271 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.339254    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 16:19:30.339391 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.339363    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 16:19:30.339603 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.339579    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 16:19:30.344126 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.344108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 16:19:30.516973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.516939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-oauth-config\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.516973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.516975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-trusted-ca-bundle\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.517175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.517002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-oauth-serving-cert\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.517175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.517088    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-console-config\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.517175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.517137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-serving-cert\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.517348 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.517178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnn8g\" (UniqueName: \"kubernetes.io/projected/be7d82c7-4e50-469d-81cb-c820a360085d-kube-api-access-hnn8g\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.517348 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.517200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-service-ca\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618352 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-console-config\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618352 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-serving-cert\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnn8g\" (UniqueName: \"kubernetes.io/projected/be7d82c7-4e50-469d-81cb-c820a360085d-kube-api-access-hnn8g\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618460    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-service-ca\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618576 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-oauth-config\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618576 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-trusted-ca-bundle\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.618576 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.618563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-oauth-serving-cert\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.619145 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.619113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-console-config\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.619337 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.619318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-trusted-ca-bundle\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.619411 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.619364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-service-ca\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.619411 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.619364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-oauth-serving-cert\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.620825 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.620802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-serving-cert\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.620925 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.620835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-oauth-config\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.627655 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.627632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnn8g\" (UniqueName: \"kubernetes.io/projected/be7d82c7-4e50-469d-81cb-c820a360085d-kube-api-access-hnn8g\") pod \"console-866488f997-8gl6s\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") " pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.644619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.644596    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:30.776454 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:30.776421    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866488f997-8gl6s"]
Jul 27 16:19:30.779658 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:19:30.779625    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7d82c7_4e50_469d_81cb_c820a360085d.slice/crio-a37720944cb37fd7052c83c8b26fe6b261e96edafe2ed40de76162a39f68fb22 WatchSource:0}: Error finding container a37720944cb37fd7052c83c8b26fe6b261e96edafe2ed40de76162a39f68fb22: Status 404 returned error can't find the container with id a37720944cb37fd7052c83c8b26fe6b261e96edafe2ed40de76162a39f68fb22
Jul 27 16:19:31.415866 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:31.415810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866488f997-8gl6s" event={"ID":"be7d82c7-4e50-469d-81cb-c820a360085d","Type":"ContainerStarted","Data":"a37720944cb37fd7052c83c8b26fe6b261e96edafe2ed40de76162a39f68fb22"}
Jul 27 16:19:34.428624 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:34.428585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866488f997-8gl6s" event={"ID":"be7d82c7-4e50-469d-81cb-c820a360085d","Type":"ContainerStarted","Data":"515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa"}
Jul 27 16:19:34.446844 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:34.446794    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866488f997-8gl6s" podStartSLOduration=1.672750798 podStartE2EDuration="4.446778469s" totalImagesPullingTime="2.774027671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:34.446197753 +0000 UTC m=+174.347338214" watchObservedRunningTime="2026-07-27 16:19:34.446778469 +0000 UTC m=+174.347918917"
Jul 27 16:19:35.399097 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:35.399067    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcfdm"
Jul 27 16:19:35.634842 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:35.634802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:35.702596 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:35.702511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:37.230244 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:37.230186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:37.537727 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:37.537554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:37.638829 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:37.638787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:37.738954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:37.738922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:40.645514 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:40.645476    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:40.645950 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:40.645580    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:40.650299 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:40.650257    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:41.459943 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:41.459916    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:19:41.597493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:41.597456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:41.655961 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:41.655927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:42.221504 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:42.221468    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:42.308575 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:42.308540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:42.556129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:42.556042    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:44.543837 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:44.543800    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:19:44.611630 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:19:44.611601    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:05.623360 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:05.623321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:05.680990 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:05.680952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:19.855266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:19.855203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:19.955194 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:19.955158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:25.872427 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:25.872387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:26.070749 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:26.070710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:26.602612 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:26.602572    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:26.710474 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:26.710440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:31.715729 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:31.715689    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:31.783863 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:31.783828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:35.656700 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:35.656656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:35.756143 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:35.756102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:35.840379 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:35.840340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:36.944935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:36.944896    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866488f997-8gl6s"]
Jul 27 16:20:37.144511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:37.144476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:37.248096 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:37.248000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:20:37.328620 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:20:37.328591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:01.963086 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:01.963039    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-866488f997-8gl6s" podUID="be7d82c7-4e50-469d-81cb-c820a360085d" containerName="console" containerID="cri-o://515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa" gracePeriod=15
Jul 27 16:21:02.193655 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.193630    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:21:02.251364 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251295    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-oauth-serving-cert\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251364 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251341    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-oauth-config\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251375    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-service-ca\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251479    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-console-config\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251524    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnn8g\" (UniqueName: \"kubernetes.io/projected/be7d82c7-4e50-469d-81cb-c820a360085d-kube-api-access-hnn8g\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251548    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-trusted-ca-bundle\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251581    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-serving-cert\") pod \"be7d82c7-4e50-469d-81cb-c820a360085d\" (UID: \"be7d82c7-4e50-469d-81cb-c820a360085d\") "
Jul 27 16:21:02.251830 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251759    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-service-ca" (OuterVolumeSpecName: "service-ca") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:21:02.251830 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251769    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:21:02.251970 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.251940    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-console-config" (OuterVolumeSpecName: "console-config") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:21:02.252065 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.252043    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:21:02.253517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.253495    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:21:02.253924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.253898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:21:02.253924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.253911    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7d82c7-4e50-469d-81cb-c820a360085d-kube-api-access-hnn8g" (OuterVolumeSpecName: "kube-api-access-hnn8g") pod "be7d82c7-4e50-469d-81cb-c820a360085d" (UID: "be7d82c7-4e50-469d-81cb-c820a360085d"). InnerVolumeSpecName "kube-api-access-hnn8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:21:02.352444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352397    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-trusted-ca-bundle\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.352444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352436    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-serving-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.352444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352446    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-oauth-serving-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.352444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352454    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be7d82c7-4e50-469d-81cb-c820a360085d-console-oauth-config\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.352687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352464    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.352687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352472    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be7d82c7-4e50-469d-81cb-c820a360085d-console-config\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.352687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.352479    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnn8g\" (UniqueName: \"kubernetes.io/projected/be7d82c7-4e50-469d-81cb-c820a360085d-kube-api-access-hnn8g\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:21:02.725583 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.725551    2559 generic.go:356] "Generic (PLEG): container finished" podID="be7d82c7-4e50-469d-81cb-c820a360085d" containerID="515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa" exitCode=0
Jul 27 16:21:02.725736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.725594    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866488f997-8gl6s"
Jul 27 16:21:02.725736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.725615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866488f997-8gl6s" event={"ID":"be7d82c7-4e50-469d-81cb-c820a360085d","Type":"ContainerDied","Data":"515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa"}
Jul 27 16:21:02.725736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.725653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866488f997-8gl6s" event={"ID":"be7d82c7-4e50-469d-81cb-c820a360085d","Type":"ContainerDied","Data":"a37720944cb37fd7052c83c8b26fe6b261e96edafe2ed40de76162a39f68fb22"}
Jul 27 16:21:02.727104 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.727088    2559 scope.go:117] "RemoveContainer" containerID="515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa"
Jul 27 16:21:02.735505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.735487    2559 scope.go:117] "RemoveContainer" containerID="515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa"
Jul 27 16:21:02.735748 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:21:02.735728    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa\": container with ID starting with 515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa not found: ID does not exist" containerID="515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa"
Jul 27 16:21:02.735822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.735755    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa"} err="failed to get container status \"515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa\": rpc error: code = NotFound desc = could not find container \"515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa\": container with ID starting with 515d1a32128c95225b0c3622fac01f883f31bafd582aa0ec8a82224e0b7380fa not found: ID does not exist"
Jul 27 16:21:02.748498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.748478    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866488f997-8gl6s"]
Jul 27 16:21:02.757088 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:02.757064    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-866488f997-8gl6s"]
Jul 27 16:21:04.734268 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:04.734235    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7d82c7-4e50-469d-81cb-c820a360085d" path="/var/lib/kubelet/pods/be7d82c7-4e50-469d-81cb-c820a360085d/volumes"
Jul 27 16:21:07.528727 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:07.528696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:07.635111 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:07.635075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:37.963825 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:37.963782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:40.494047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:40.494019    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 16:21:46.196902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:46.196862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:46.288864 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:46.288825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:53.048520 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:53.048485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:21:54.240227 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:21:54.240195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:13.268760 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:13.268721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:13.699638 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:13.699609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:17.139334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:17.139303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:17.234246 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:17.234209    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:18.672424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:18.672374    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:18.793247 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:18.793213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:19.431881 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:19.431817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:19.625045 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:19.625011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:21.793425 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:21.793389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:22.073431 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:22.073332    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:23.705960 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:23.705909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:23.808660 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:23.808629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:27.101097 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.101065    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qrvq9"]
Jul 27 16:22:27.101473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.101296    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7d82c7-4e50-469d-81cb-c820a360085d" containerName="console"
Jul 27 16:22:27.101473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.101307    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7d82c7-4e50-469d-81cb-c820a360085d" containerName="console"
Jul 27 16:22:27.101473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.101346    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7d82c7-4e50-469d-81cb-c820a360085d" containerName="console"
Jul 27 16:22:27.103188 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.103173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.106076 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.106054    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 16:22:27.107227 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.107209    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 16:22:27.107227 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.107217    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fg4pc\""
Jul 27 16:22:27.119519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.119499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qrvq9"]
Jul 27 16:22:27.226968 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.226929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfx9b\" (UniqueName: \"kubernetes.io/projected/f8db5d82-a650-4395-93c4-bc3ce7e80b30-kube-api-access-jfx9b\") pod \"cert-manager-webhook-6486ffd997-qrvq9\" (UID: \"f8db5d82-a650-4395-93c4-bc3ce7e80b30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.227144 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.226977    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8db5d82-a650-4395-93c4-bc3ce7e80b30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qrvq9\" (UID: \"f8db5d82-a650-4395-93c4-bc3ce7e80b30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.328275 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.328224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfx9b\" (UniqueName: \"kubernetes.io/projected/f8db5d82-a650-4395-93c4-bc3ce7e80b30-kube-api-access-jfx9b\") pod \"cert-manager-webhook-6486ffd997-qrvq9\" (UID: \"f8db5d82-a650-4395-93c4-bc3ce7e80b30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.328410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.328315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8db5d82-a650-4395-93c4-bc3ce7e80b30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qrvq9\" (UID: \"f8db5d82-a650-4395-93c4-bc3ce7e80b30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.346431 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.346399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8db5d82-a650-4395-93c4-bc3ce7e80b30-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qrvq9\" (UID: \"f8db5d82-a650-4395-93c4-bc3ce7e80b30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.347061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.347041    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfx9b\" (UniqueName: \"kubernetes.io/projected/f8db5d82-a650-4395-93c4-bc3ce7e80b30-kube-api-access-jfx9b\") pod \"cert-manager-webhook-6486ffd997-qrvq9\" (UID: \"f8db5d82-a650-4395-93c4-bc3ce7e80b30\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.411717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.411687    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:27.537006 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.536974    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qrvq9"]
Jul 27 16:22:27.540012 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:22:27.539981    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8db5d82_a650_4395_93c4_bc3ce7e80b30.slice/crio-f9ef333ad639e84f3cdf778db1ccc2666028250482322eccbd3388c01b6abd30 WatchSource:0}: Error finding container f9ef333ad639e84f3cdf778db1ccc2666028250482322eccbd3388c01b6abd30: Status 404 returned error can't find the container with id f9ef333ad639e84f3cdf778db1ccc2666028250482322eccbd3388c01b6abd30
Jul 27 16:22:27.541962 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:27.541945    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:22:28.003473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:28.003441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9" event={"ID":"f8db5d82-a650-4395-93c4-bc3ce7e80b30","Type":"ContainerStarted","Data":"f9ef333ad639e84f3cdf778db1ccc2666028250482322eccbd3388c01b6abd30"}
Jul 27 16:22:29.402110 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:29.402078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:30.780957 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:30.780877    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:30.869844 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:30.869809    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:31.016866 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:31.016832    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9" event={"ID":"f8db5d82-a650-4395-93c4-bc3ce7e80b30","Type":"ContainerStarted","Data":"cffc2e980ad8f2be33ff95686e9fd87650b23accf1c0bbaa5a2b773e13a1dcec"}
Jul 27 16:22:31.017022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:31.016948    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:31.046708 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:31.046607    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9" podStartSLOduration=1.068301675 podStartE2EDuration="4.046592204s" totalImagesPullingTime="2.978290529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:22:31.045708509 +0000 UTC m=+350.946848957" watchObservedRunningTime="2026-07-27 16:22:31.046592204 +0000 UTC m=+350.947732651"
Jul 27 16:22:34.716691 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:34.716605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:35.503317 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:35.503262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:37.022430 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:37.022400    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qrvq9"
Jul 27 16:22:41.190068 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:41.190028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:41.288840 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:41.288794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:42.998484 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:42.998448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:58.065326 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:58.065267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:58.223924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:58.223880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:58.917947 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:58.917907    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:59.024348 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.024316    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:59.121223 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.121189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:59.166946 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.166913    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"]
Jul 27 16:22:59.169017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.168978    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.174864 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.174836    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 16:22:59.175495 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.175475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 16:22:59.175719 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.175704    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 16:22:59.175987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.175967    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 16:22:59.186229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.186211    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jgz5w\""
Jul 27 16:22:59.265211 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.265176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0f648ea-5889-4b82-96da-5d897d07f9b2-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.265211 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.265219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f648ea-5889-4b82-96da-5d897d07f9b2-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.265464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.265246    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/e0f648ea-5889-4b82-96da-5d897d07f9b2-kube-api-access-8kvwr\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.265464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.265369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0f648ea-5889-4b82-96da-5d897d07f9b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.265464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.265394    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0f648ea-5889-4b82-96da-5d897d07f9b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.284818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.284784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"]
Jul 27 16:22:59.365914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.365864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0f648ea-5889-4b82-96da-5d897d07f9b2-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.365914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.365913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f648ea-5889-4b82-96da-5d897d07f9b2-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.366148 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.366028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/e0f648ea-5889-4b82-96da-5d897d07f9b2-kube-api-access-8kvwr\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.366148 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.366129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0f648ea-5889-4b82-96da-5d897d07f9b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.366242 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.366161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0f648ea-5889-4b82-96da-5d897d07f9b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.366324 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.366247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0f648ea-5889-4b82-96da-5d897d07f9b2-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.368323 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.368294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f648ea-5889-4b82-96da-5d897d07f9b2-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.368478 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.368463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0f648ea-5889-4b82-96da-5d897d07f9b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.368519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.368488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0f648ea-5889-4b82-96da-5d897d07f9b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.383096 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.383075    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvwr\" (UniqueName: \"kubernetes.io/projected/e0f648ea-5889-4b82-96da-5d897d07f9b2-kube-api-access-8kvwr\") pod \"opendatahub-operator-controller-manager-55548c94b5-rdlh6\" (UID: \"e0f648ea-5889-4b82-96da-5d897d07f9b2\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.383096 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.383082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:59.479222 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.479131    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:22:59.615635 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.615605    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"]
Jul 27 16:22:59.619588 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:22:59.619561    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f648ea_5889_4b82_96da_5d897d07f9b2.slice/crio-e8c26c9f5f0d50d83a44019c65125428c621970c97f232d8202474d6da718a76 WatchSource:0}: Error finding container e8c26c9f5f0d50d83a44019c65125428c621970c97f232d8202474d6da718a76: Status 404 returned error can't find the container with id e8c26c9f5f0d50d83a44019c65125428c621970c97f232d8202474d6da718a76
Jul 27 16:22:59.679492 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.679461    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:59.782075 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.781984    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:22:59.880260 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:22:59.880225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:00.111665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:00.111579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6" event={"ID":"e0f648ea-5889-4b82-96da-5d897d07f9b2","Type":"ContainerStarted","Data":"e8c26c9f5f0d50d83a44019c65125428c621970c97f232d8202474d6da718a76"}
Jul 27 16:23:01.341692 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:01.341647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:01.448829 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:01.448793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:03.123580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:03.123541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6" event={"ID":"e0f648ea-5889-4b82-96da-5d897d07f9b2","Type":"ContainerStarted","Data":"689262169cf248a56ce80636c9b65c8ce706ba9eed3251127c31bc5370695292"}
Jul 27 16:23:04.127876 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:04.127845    2559 generic.go:356] "Generic (PLEG): container finished" podID="e0f648ea-5889-4b82-96da-5d897d07f9b2" containerID="689262169cf248a56ce80636c9b65c8ce706ba9eed3251127c31bc5370695292" exitCode=0
Jul 27 16:23:04.128213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:04.127890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6" event={"ID":"e0f648ea-5889-4b82-96da-5d897d07f9b2","Type":"ContainerDied","Data":"689262169cf248a56ce80636c9b65c8ce706ba9eed3251127c31bc5370695292"}
Jul 27 16:23:05.132988 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:05.132952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6" event={"ID":"e0f648ea-5889-4b82-96da-5d897d07f9b2","Type":"ContainerStarted","Data":"1c064cc8c1befc0661c600b4ed85495a9321f738e87f189fcf96a138be1dfcf8"}
Jul 27 16:23:05.133390 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:05.133025    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:23:05.164760 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:05.164697    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6" podStartSLOduration=1.355015098 podStartE2EDuration="6.164675341s" totalImagesPullingTime="3.785311629s" totalInitContainerRuntime="1.024348614s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:05.162672591 +0000 UTC m=+385.063813062" watchObservedRunningTime="2026-07-27 16:23:05.164675341 +0000 UTC m=+385.065815792"
Jul 27 16:23:16.138416 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:16.138387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-rdlh6"
Jul 27 16:23:18.490488 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:18.490451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:18.636401 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:18.636360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.059815 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.059781    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zjvv4"]
Jul 27 16:23:19.065724 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.065706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.069507 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.069482    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5g2vl\""
Jul 27 16:23:19.069754 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.069725    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 27 16:23:19.075623 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.075592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zjvv4"]
Jul 27 16:23:19.108051 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.108023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.182910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.182867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.218952 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.218867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46619dce-d9fc-47b2-b8b7-d52ae6875b81-cert\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.218952 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.218913    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8g9\" (UniqueName: \"kubernetes.io/projected/46619dce-d9fc-47b2-b8b7-d52ae6875b81-kube-api-access-tt8g9\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.287001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.286958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.320214 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.320171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46619dce-d9fc-47b2-b8b7-d52ae6875b81-cert\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.320394 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.320320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8g9\" (UniqueName: \"kubernetes.io/projected/46619dce-d9fc-47b2-b8b7-d52ae6875b81-kube-api-access-tt8g9\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.320394 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:23:19.320333    2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 16:23:19.320506 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:23:19.320398    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46619dce-d9fc-47b2-b8b7-d52ae6875b81-cert podName:46619dce-d9fc-47b2-b8b7-d52ae6875b81 nodeName:}" failed. No retries permitted until 2026-07-27 16:23:19.820380764 +0000 UTC m=+399.721521190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/46619dce-d9fc-47b2-b8b7-d52ae6875b81-cert") pod "odh-model-controller-7998c67dc4-zjvv4" (UID: "46619dce-d9fc-47b2-b8b7-d52ae6875b81") : secret "odh-model-controller-webhook-cert" not found
Jul 27 16:23:19.330992 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.330965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8g9\" (UniqueName: \"kubernetes.io/projected/46619dce-d9fc-47b2-b8b7-d52ae6875b81-kube-api-access-tt8g9\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.487537 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.487450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.607794 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.607763    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g7k8c"]
Jul 27 16:23:19.609838 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.609819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.616920 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.616899    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 27 16:23:19.625563 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.625541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g7k8c"]
Jul 27 16:23:19.635896 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.635857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.723797 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.723763    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13114416-d6a5-484c-99b2-674b29cf37ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-g7k8c\" (UID: \"13114416-d6a5-484c-99b2-674b29cf37ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.723797 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.723798    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mj9\" (UniqueName: \"kubernetes.io/projected/13114416-d6a5-484c-99b2-674b29cf37ea-kube-api-access-58mj9\") pod \"model-serving-api-54d89bd5fc-g7k8c\" (UID: \"13114416-d6a5-484c-99b2-674b29cf37ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.763049 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.762943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:19.824662 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.824630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46619dce-d9fc-47b2-b8b7-d52ae6875b81-cert\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.824822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.824685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13114416-d6a5-484c-99b2-674b29cf37ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-g7k8c\" (UID: \"13114416-d6a5-484c-99b2-674b29cf37ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.824822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.824713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mj9\" (UniqueName: \"kubernetes.io/projected/13114416-d6a5-484c-99b2-674b29cf37ea-kube-api-access-58mj9\") pod \"model-serving-api-54d89bd5fc-g7k8c\" (UID: \"13114416-d6a5-484c-99b2-674b29cf37ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.827123 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.827093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13114416-d6a5-484c-99b2-674b29cf37ea-tls-certs\") pod \"model-serving-api-54d89bd5fc-g7k8c\" (UID: \"13114416-d6a5-484c-99b2-674b29cf37ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.827238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.827098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46619dce-d9fc-47b2-b8b7-d52ae6875b81-cert\") pod \"odh-model-controller-7998c67dc4-zjvv4\" (UID: \"46619dce-d9fc-47b2-b8b7-d52ae6875b81\") " pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:19.840711 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.840678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mj9\" (UniqueName: \"kubernetes.io/projected/13114416-d6a5-484c-99b2-674b29cf37ea-kube-api-access-58mj9\") pod \"model-serving-api-54d89bd5fc-g7k8c\" (UID: \"13114416-d6a5-484c-99b2-674b29cf37ea\") " pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.920137 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.920100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:19.977997 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:19.977962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:20.052682 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:20.052651    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g7k8c"]
Jul 27 16:23:20.056005 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:23:20.055974    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13114416_d6a5_484c_99b2_674b29cf37ea.slice/crio-fd4421d7d4c6d3dea9804d6c579c064824391136a1b429a8a7e47580ffe6ff4c WatchSource:0}: Error finding container fd4421d7d4c6d3dea9804d6c579c064824391136a1b429a8a7e47580ffe6ff4c: Status 404 returned error can't find the container with id fd4421d7d4c6d3dea9804d6c579c064824391136a1b429a8a7e47580ffe6ff4c
Jul 27 16:23:20.119440 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:20.119409    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zjvv4"]
Jul 27 16:23:20.122681 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:23:20.122649    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46619dce_d9fc_47b2_b8b7_d52ae6875b81.slice/crio-13b5bc130bc6055bde7e82b11a465fb8a6f8dc427151c33f4796a67ea9f5dd7e WatchSource:0}: Error finding container 13b5bc130bc6055bde7e82b11a465fb8a6f8dc427151c33f4796a67ea9f5dd7e: Status 404 returned error can't find the container with id 13b5bc130bc6055bde7e82b11a465fb8a6f8dc427151c33f4796a67ea9f5dd7e
Jul 27 16:23:20.185036 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:20.184994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4" event={"ID":"46619dce-d9fc-47b2-b8b7-d52ae6875b81","Type":"ContainerStarted","Data":"13b5bc130bc6055bde7e82b11a465fb8a6f8dc427151c33f4796a67ea9f5dd7e"}
Jul 27 16:23:20.186169 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:20.186143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c" event={"ID":"13114416-d6a5-484c-99b2-674b29cf37ea","Type":"ContainerStarted","Data":"fd4421d7d4c6d3dea9804d6c579c064824391136a1b429a8a7e47580ffe6ff4c"}
Jul 27 16:23:21.807732 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:21.807669    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:21.920858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:21.920822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:23.063516 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:23.063458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:23.131406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:23.131365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:23.201829 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:23.201791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4" event={"ID":"46619dce-d9fc-47b2-b8b7-d52ae6875b81","Type":"ContainerStarted","Data":"7cea3d2ac79117d3443a32d36c94046950a865105224883b2b9254a75d6649a0"}
Jul 27 16:23:23.202017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:23.201909    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:23.223172 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:23.223116    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4" podStartSLOduration=1.318041878 podStartE2EDuration="4.22309368s" totalImagesPullingTime="2.905051802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:23.220423206 +0000 UTC m=+403.121563653" watchObservedRunningTime="2026-07-27 16:23:23.22309368 +0000 UTC m=+403.124234130"
Jul 27 16:23:24.137261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.137224    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:24.567303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.567250    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"]
Jul 27 16:23:24.569308 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.569271    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.575367 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.575342    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 16:23:24.575519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.575391    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 16:23:24.575519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.575398    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:23:24.575519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.575463    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 16:23:24.575803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.575667    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 16:23:24.575872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.575835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4gzbz\""
Jul 27 16:23:24.579685 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.579662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"]
Jul 27 16:23:24.662638 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.662553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc0559f6-dc1d-48a7-a18a-2c8613af601d-metrics-cert\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.662638 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.662600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844kd\" (UniqueName: \"kubernetes.io/projected/bc0559f6-dc1d-48a7-a18a-2c8613af601d-kube-api-access-844kd\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.662638 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.662629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0559f6-dc1d-48a7-a18a-2c8613af601d-cert\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.662850 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.662653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc0559f6-dc1d-48a7-a18a-2c8613af601d-manager-config\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.763112 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.763064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844kd\" (UniqueName: \"kubernetes.io/projected/bc0559f6-dc1d-48a7-a18a-2c8613af601d-kube-api-access-844kd\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.763112 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.763112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0559f6-dc1d-48a7-a18a-2c8613af601d-cert\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.763407 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.763145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc0559f6-dc1d-48a7-a18a-2c8613af601d-manager-config\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.763407 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.763240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc0559f6-dc1d-48a7-a18a-2c8613af601d-metrics-cert\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.764050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.764021    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc0559f6-dc1d-48a7-a18a-2c8613af601d-manager-config\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.765746 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.765719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc0559f6-dc1d-48a7-a18a-2c8613af601d-metrics-cert\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.765995 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.765972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0559f6-dc1d-48a7-a18a-2c8613af601d-cert\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.772755 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.772731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-844kd\" (UniqueName: \"kubernetes.io/projected/bc0559f6-dc1d-48a7-a18a-2c8613af601d-kube-api-access-844kd\") pod \"lws-controller-manager-cd69fdcb8-hk48f\" (UID: \"bc0559f6-dc1d-48a7-a18a-2c8613af601d\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:24.878949 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:24.878909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:25.008921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:25.008881    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"]
Jul 27 16:23:25.011487 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:23:25.011461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0559f6_dc1d_48a7_a18a_2c8613af601d.slice/crio-796cb22124934f84a239714f8303ee8ec7962e85e0f61ee5a552c6e059e01383 WatchSource:0}: Error finding container 796cb22124934f84a239714f8303ee8ec7962e85e0f61ee5a552c6e059e01383: Status 404 returned error can't find the container with id 796cb22124934f84a239714f8303ee8ec7962e85e0f61ee5a552c6e059e01383
Jul 27 16:23:25.211710 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:25.211613    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f" event={"ID":"bc0559f6-dc1d-48a7-a18a-2c8613af601d","Type":"ContainerStarted","Data":"796cb22124934f84a239714f8303ee8ec7962e85e0f61ee5a552c6e059e01383"}
Jul 27 16:23:25.213272 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:25.213244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c" event={"ID":"13114416-d6a5-484c-99b2-674b29cf37ea","Type":"ContainerStarted","Data":"941fe0671faf937bf1294e9668d933c1c00592aad15013ad630bc15262f4eced"}
Jul 27 16:23:25.213517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:25.213499    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:25.251430 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:25.251369    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c" podStartSLOduration=1.9449681920000002 podStartE2EDuration="6.2513474s" totalImagesPullingTime="4.306379208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:25.250973036 +0000 UTC m=+405.152113484" watchObservedRunningTime="2026-07-27 16:23:25.2513474 +0000 UTC m=+405.152487848"
Jul 27 16:23:27.021298 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.021240    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"]
Jul 27 16:23:27.027486 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.027461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.031836 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.031785    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 16:23:27.031976 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.031846    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 16:23:27.032586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.032345    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 16:23:27.032586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.032385    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 16:23:27.032586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.032406    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 16:23:27.032586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.032447    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 16:23:27.032586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.032407    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mbr74\""
Jul 27 16:23:27.039540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.039515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"]
Jul 27 16:23:27.078249 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078216    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.078415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.078415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzkj\" (UniqueName: \"kubernetes.io/projected/56f0e6ee-5cb5-41a4-b05a-31af48de697f-kube-api-access-ftzkj\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.078415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.078415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078395    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.078602 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0e6ee-5cb5-41a4-b05a-31af48de697f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.078602 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.078470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179079 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0e6ee-5cb5-41a4-b05a-31af48de697f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179252 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179340 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179340 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179503 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzkj\" (UniqueName: \"kubernetes.io/projected/56f0e6ee-5cb5-41a4-b05a-31af48de697f-kube-api-access-ftzkj\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179503 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.179503 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.179425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.180117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.180066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.182019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.181977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.182131 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.182083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.182190 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.182176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.182331 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.182315    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f0e6ee-5cb5-41a4-b05a-31af48de697f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.189621 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.189577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzkj\" (UniqueName: \"kubernetes.io/projected/56f0e6ee-5cb5-41a4-b05a-31af48de697f-kube-api-access-ftzkj\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.192034 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.191983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f0e6ee-5cb5-41a4-b05a-31af48de697f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-98x86\" (UID: \"56f0e6ee-5cb5-41a4-b05a-31af48de697f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.339443 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.339354    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:27.789489 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.789463    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"]
Jul 27 16:23:27.791456 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:23:27.791430    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f0e6ee_5cb5_41a4_b05a_31af48de697f.slice/crio-0d78356c6505dd551582c175385db62529af8cf9f8d1df7ea38f8116b4811bcd WatchSource:0}: Error finding container 0d78356c6505dd551582c175385db62529af8cf9f8d1df7ea38f8116b4811bcd: Status 404 returned error can't find the container with id 0d78356c6505dd551582c175385db62529af8cf9f8d1df7ea38f8116b4811bcd
Jul 27 16:23:27.839942 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.839899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:27.951937 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:27.951880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:28.228916 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:28.228831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f" event={"ID":"bc0559f6-dc1d-48a7-a18a-2c8613af601d","Type":"ContainerStarted","Data":"00954bae5a2b4013b03b5741670bd39ece4429a0e596e35c069429244d3b961f"}
Jul 27 16:23:28.229366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:28.228981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:23:28.230109 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:28.230088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86" event={"ID":"56f0e6ee-5cb5-41a4-b05a-31af48de697f","Type":"ContainerStarted","Data":"0d78356c6505dd551582c175385db62529af8cf9f8d1df7ea38f8116b4811bcd"}
Jul 27 16:23:28.256092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:28.256045    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f" podStartSLOduration=1.595353513 podStartE2EDuration="4.256030836s" totalImagesPullingTime="2.660677323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:28.255946986 +0000 UTC m=+408.157087433" watchObservedRunningTime="2026-07-27 16:23:28.256030836 +0000 UTC m=+408.157171284"
Jul 27 16:23:29.224062 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:29.224012    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:29.481362 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:29.481245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:29.580131 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:29.580087    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:30.452459 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:30.452422    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:23:30.452553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:30.452495    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:23:30.958329 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:30.958267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:31.090100 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.090068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:31.243641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.243549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86" event={"ID":"56f0e6ee-5cb5-41a4-b05a-31af48de697f","Type":"ContainerStarted","Data":"59e34d6ed7a422b5d76f12339034b668ccc7938b773bca3aef87dcbf13f2e9b3"}
Jul 27 16:23:31.243806 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.243756    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:31.245340 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.245317    2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-98x86 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 16:23:31.245451 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.245361    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86" podUID="56f0e6ee-5cb5-41a4-b05a-31af48de697f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 16:23:31.445453 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.445415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:31.524969 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:31.524887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:32.247146 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:32.247110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:32.248491 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:32.248467    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86"
Jul 27 16:23:32.297617 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:32.297546    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-98x86" podStartSLOduration=3.638709123 podStartE2EDuration="6.297530677s" totalImagesPullingTime="2.658821554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:31.288238325 +0000 UTC m=+411.189378773" watchObservedRunningTime="2026-07-27 16:23:32.297530677 +0000 UTC m=+412.198671121"
Jul 27 16:23:32.334767 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:32.334735    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:23:34.208312 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:34.208266    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zjvv4"
Jul 27 16:23:36.222018 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:36.221987    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-g7k8c"
Jul 27 16:23:39.237772 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:23:39.237738    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-hk48f"
Jul 27 16:24:19.839013 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:19.838974    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:23.877678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:23.877642    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:48.111020 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:48.110983    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:48.709304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:48.709251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:49.763932 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:49.763901    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:49.859986 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:49.859935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:52.740386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:52.740350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:55.360222 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:55.360186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:55.450675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:55.450643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:56.510977 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:56.510941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:58.520749 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:58.520710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:24:58.784140 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:24:58.784053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:01.173567 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:01.173529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:02.322364 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:02.322328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:02.423138 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:02.423084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:03.078963 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:03.078929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:03.174940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:03.174907    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:03.426139 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:03.426109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:03.688104 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:03.688016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:03.800138 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:03.800106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:06.553097 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:06.553065    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:06.734520 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:06.734481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:07.054454 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:07.054410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:07.214899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:07.214855    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:07.623517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:07.623475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:07.738848 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:07.738819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:14.741988 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:14.741944    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:14.854129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:14.854092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:19.077958 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:19.077918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:19.185440 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:19.185393    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:23.127018 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:23.126987    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:23.227462 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:23.227422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:23.732144 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:23.732110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:23.833294 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:23.833251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:27.724239 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:27.724200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:27.822325 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:27.822263    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:31.857985 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:31.857948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:34.011556 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:34.011471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:34.235640 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:34.235606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:38.760176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:38.760138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:38.862368 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:38.862339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:45.919198 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:45.919162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:46.028261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:46.028219    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:46.980843 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:46.980810    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:47.082941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:47.082903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:49.484601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:49.484566    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:49.588299 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:49.588246    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:53.027608 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:53.027574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:53.092717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:53.092685    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:57.832487 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:57.832447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:25:57.933029 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:25:57.932992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:04.462409 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:04.462348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:04.768011 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:04.767917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:04.881090 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:04.881053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:04.948678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:04.948648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:05.709015 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:05.708982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:05.814470 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:05.814417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:22.703792 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:22.703741    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:22.803974 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:22.803936    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:23.184993 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:23.184948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:35.184816 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:35.184742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:35.328814 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:35.328771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:35.705918 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:35.705881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:42.608701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.608666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:42.633496 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.633467    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76849c8c77-htjmf"]
Jul 27 16:26:42.635663 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.635643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.638896 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.638874    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wn5rq\""
Jul 27 16:26:42.639673 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.639656    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 16:26:42.639757 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.639658    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 16:26:42.649630 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.649606    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76849c8c77-htjmf"]
Jul 27 16:26:42.740068 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.740040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0367179-076e-4683-8143-2ddc9304cf26-webhook-cert\") pod \"maas-controller-76849c8c77-htjmf\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") " pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.740068 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.740072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2r2\" (UniqueName: \"kubernetes.io/projected/d0367179-076e-4683-8143-2ddc9304cf26-kube-api-access-7w2r2\") pod \"maas-controller-76849c8c77-htjmf\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") " pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.820233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.820171    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-556f76c959-5r57l"]
Jul 27 16:26:42.822404 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.822381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:42.835195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.835172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556f76c959-5r57l"]
Jul 27 16:26:42.841250 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.841223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0367179-076e-4683-8143-2ddc9304cf26-webhook-cert\") pod \"maas-controller-76849c8c77-htjmf\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") " pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.841391 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.841292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2r2\" (UniqueName: \"kubernetes.io/projected/d0367179-076e-4683-8143-2ddc9304cf26-kube-api-access-7w2r2\") pod \"maas-controller-76849c8c77-htjmf\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") " pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.843747 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.843723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0367179-076e-4683-8143-2ddc9304cf26-webhook-cert\") pod \"maas-controller-76849c8c77-htjmf\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") " pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.852872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.852851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2r2\" (UniqueName: \"kubernetes.io/projected/d0367179-076e-4683-8143-2ddc9304cf26-kube-api-access-7w2r2\") pod \"maas-controller-76849c8c77-htjmf\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") " pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.942455 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.942423    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-webhook-cert\") pod \"maas-controller-556f76c959-5r57l\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") " pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:42.942643 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.942468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7598\" (UniqueName: \"kubernetes.io/projected/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-kube-api-access-t7598\") pod \"maas-controller-556f76c959-5r57l\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") " pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:42.946333 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.946302    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:42.953591 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:42.953544    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:43.042869 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.042840    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-webhook-cert\") pod \"maas-controller-556f76c959-5r57l\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") " pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:43.043053 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.042886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7598\" (UniqueName: \"kubernetes.io/projected/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-kube-api-access-t7598\") pod \"maas-controller-556f76c959-5r57l\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") " pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:43.045602 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.045578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-webhook-cert\") pod \"maas-controller-556f76c959-5r57l\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") " pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:43.052682 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.052652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7598\" (UniqueName: \"kubernetes.io/projected/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-kube-api-access-t7598\") pod \"maas-controller-556f76c959-5r57l\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") " pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:43.078499 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.078468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76849c8c77-htjmf"]
Jul 27 16:26:43.082293 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:26:43.082253    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0367179_076e_4683_8143_2ddc9304cf26.slice/crio-76329ba19af8e1eb9ba6d03ffba4ba540f6399710d8fe96193e61da6c2f27304 WatchSource:0}: Error finding container 76329ba19af8e1eb9ba6d03ffba4ba540f6399710d8fe96193e61da6c2f27304: Status 404 returned error can't find the container with id 76329ba19af8e1eb9ba6d03ffba4ba540f6399710d8fe96193e61da6c2f27304
Jul 27 16:26:43.118722 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.118686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:43.132496 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.132469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:43.262988 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.262953    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556f76c959-5r57l"]
Jul 27 16:26:43.265878 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:26:43.265854    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92eba6f2_c76e_45a1_80d1_6fc2b0842b83.slice/crio-c8559a82ef4eba87193132bdc6bba3b717359f66df48c14ebfde0e8000efda87 WatchSource:0}: Error finding container c8559a82ef4eba87193132bdc6bba3b717359f66df48c14ebfde0e8000efda87: Status 404 returned error can't find the container with id c8559a82ef4eba87193132bdc6bba3b717359f66df48c14ebfde0e8000efda87
Jul 27 16:26:43.945077 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.945034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-5r57l" event={"ID":"92eba6f2-c76e-45a1-80d1-6fc2b0842b83","Type":"ContainerStarted","Data":"c8559a82ef4eba87193132bdc6bba3b717359f66df48c14ebfde0e8000efda87"}
Jul 27 16:26:43.946961 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:43.946897    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76849c8c77-htjmf" event={"ID":"d0367179-076e-4683-8143-2ddc9304cf26","Type":"ContainerStarted","Data":"76329ba19af8e1eb9ba6d03ffba4ba540f6399710d8fe96193e61da6c2f27304"}
Jul 27 16:26:45.796266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.796223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:45.885032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.884940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:26:45.957828 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.957792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76849c8c77-htjmf" event={"ID":"d0367179-076e-4683-8143-2ddc9304cf26","Type":"ContainerStarted","Data":"2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604"}
Jul 27 16:26:45.958024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.957908    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:45.959315 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.959259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-5r57l" event={"ID":"92eba6f2-c76e-45a1-80d1-6fc2b0842b83","Type":"ContainerStarted","Data":"77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c"}
Jul 27 16:26:45.959442 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.959334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:45.981561 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:45.981500    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76849c8c77-htjmf" podStartSLOduration=1.4492207449999999 podStartE2EDuration="3.981476501s" totalImagesPullingTime="2.532255756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:45.979927378 +0000 UTC m=+605.881067826" watchObservedRunningTime="2026-07-27 16:26:45.981476501 +0000 UTC m=+605.882616950"
Jul 27 16:26:46.002201 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:46.002149    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-556f76c959-5r57l" podStartSLOduration=1.6489696729999999 podStartE2EDuration="4.002133849s" totalImagesPullingTime="2.353164176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:46.000405412 +0000 UTC m=+605.901545862" watchObservedRunningTime="2026-07-27 16:26:46.002133849 +0000 UTC m=+605.903274297"
Jul 27 16:26:56.965664 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:56.965631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:26:56.967299 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:56.965702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:57.009102 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.009068    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76849c8c77-htjmf"]
Jul 27 16:26:57.009337 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.009294    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76849c8c77-htjmf" podUID="d0367179-076e-4683-8143-2ddc9304cf26" containerName="manager" containerID="cri-o://2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604" gracePeriod=30
Jul 27 16:26:57.249817 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.249794    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:57.364066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.364025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w2r2\" (UniqueName: \"kubernetes.io/projected/d0367179-076e-4683-8143-2ddc9304cf26-kube-api-access-7w2r2\") pod \"d0367179-076e-4683-8143-2ddc9304cf26\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") "
Jul 27 16:26:57.364232 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.364158    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0367179-076e-4683-8143-2ddc9304cf26-webhook-cert\") pod \"d0367179-076e-4683-8143-2ddc9304cf26\" (UID: \"d0367179-076e-4683-8143-2ddc9304cf26\") "
Jul 27 16:26:57.366141 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.366115    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0367179-076e-4683-8143-2ddc9304cf26-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d0367179-076e-4683-8143-2ddc9304cf26" (UID: "d0367179-076e-4683-8143-2ddc9304cf26"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:26:57.366236 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.366221    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0367179-076e-4683-8143-2ddc9304cf26-kube-api-access-7w2r2" (OuterVolumeSpecName: "kube-api-access-7w2r2") pod "d0367179-076e-4683-8143-2ddc9304cf26" (UID: "d0367179-076e-4683-8143-2ddc9304cf26"). InnerVolumeSpecName "kube-api-access-7w2r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:26:57.464771 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.464726    2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0367179-076e-4683-8143-2ddc9304cf26-webhook-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:26:57.464771 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:57.464764    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w2r2\" (UniqueName: \"kubernetes.io/projected/d0367179-076e-4683-8143-2ddc9304cf26-kube-api-access-7w2r2\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:26:58.013827 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.013779    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0367179-076e-4683-8143-2ddc9304cf26" containerID="2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604" exitCode=0
Jul 27 16:26:58.013827 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.013827    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76849c8c77-htjmf"
Jul 27 16:26:58.014408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.013857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76849c8c77-htjmf" event={"ID":"d0367179-076e-4683-8143-2ddc9304cf26","Type":"ContainerDied","Data":"2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604"}
Jul 27 16:26:58.014408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.013914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76849c8c77-htjmf" event={"ID":"d0367179-076e-4683-8143-2ddc9304cf26","Type":"ContainerDied","Data":"76329ba19af8e1eb9ba6d03ffba4ba540f6399710d8fe96193e61da6c2f27304"}
Jul 27 16:26:58.014408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.013936    2559 scope.go:117] "RemoveContainer" containerID="2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604"
Jul 27 16:26:58.022497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.022475    2559 scope.go:117] "RemoveContainer" containerID="2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604"
Jul 27 16:26:58.022756 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:26:58.022740    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604\": container with ID starting with 2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604 not found: ID does not exist" containerID="2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604"
Jul 27 16:26:58.022801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.022764    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604"} err="failed to get container status \"2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604\": rpc error: code = NotFound desc = could not find container \"2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604\": container with ID starting with 2db88b26d91e40e8d1aa6c472e0abe35472a16534109bb78a6d13755448e4604 not found: ID does not exist"
Jul 27 16:26:58.036290 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.036249    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76849c8c77-htjmf"]
Jul 27 16:26:58.043502 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.043479    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76849c8c77-htjmf"]
Jul 27 16:26:58.735477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:26:58.735443    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0367179-076e-4683-8143-2ddc9304cf26" path="/var/lib/kubelet/pods/d0367179-076e-4683-8143-2ddc9304cf26/volumes"
Jul 27 16:27:17.592938 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:17.592885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:17.678327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:17.678262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:18.061290 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.061239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:18.169092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.169057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:18.653781 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.653739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:18.765160 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.765115    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:18.805736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.805700    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-qxpr8"]
Jul 27 16:27:18.806058 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.806039    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0367179-076e-4683-8143-2ddc9304cf26" containerName="manager"
Jul 27 16:27:18.806138 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.806061    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0367179-076e-4683-8143-2ddc9304cf26" containerName="manager"
Jul 27 16:27:18.806190 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.806147    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0367179-076e-4683-8143-2ddc9304cf26" containerName="manager"
Jul 27 16:27:18.816606 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.816578    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:18.820382 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.820353    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 16:27:18.820495 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.820415    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bqx2k\""
Jul 27 16:27:18.827043 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.827019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-qxpr8"]
Jul 27 16:27:18.935485 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.935407    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6nf\" (UniqueName: \"kubernetes.io/projected/2e603be2-8915-481b-9db0-4bf942836c39-kube-api-access-jn6nf\") pod \"payload-processing-5cc6cd444b-qxpr8\" (UID: \"2e603be2-8915-481b-9db0-4bf942836c39\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:18.935485 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:18.935452    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e603be2-8915-481b-9db0-4bf942836c39-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-qxpr8\" (UID: \"2e603be2-8915-481b-9db0-4bf942836c39\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:19.036894 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:19.036855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6nf\" (UniqueName: \"kubernetes.io/projected/2e603be2-8915-481b-9db0-4bf942836c39-kube-api-access-jn6nf\") pod \"payload-processing-5cc6cd444b-qxpr8\" (UID: \"2e603be2-8915-481b-9db0-4bf942836c39\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:19.036894 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:19.036896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e603be2-8915-481b-9db0-4bf942836c39-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-qxpr8\" (UID: \"2e603be2-8915-481b-9db0-4bf942836c39\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:19.037598 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:19.037578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e603be2-8915-481b-9db0-4bf942836c39-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-qxpr8\" (UID: \"2e603be2-8915-481b-9db0-4bf942836c39\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:19.047243 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:19.047213    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6nf\" (UniqueName: \"kubernetes.io/projected/2e603be2-8915-481b-9db0-4bf942836c39-kube-api-access-jn6nf\") pod \"payload-processing-5cc6cd444b-qxpr8\" (UID: \"2e603be2-8915-481b-9db0-4bf942836c39\") " pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:19.127858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:19.127819    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:19.305188 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:19.305164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-qxpr8"]
Jul 27 16:27:19.308017 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:27:19.307989    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e603be2_8915_481b_9db0_4bf942836c39.slice/crio-50fd4a837d1827c0ea6d7d471e707f8c43db11e9a48190defcb7c9a5d8b21c74 WatchSource:0}: Error finding container 50fd4a837d1827c0ea6d7d471e707f8c43db11e9a48190defcb7c9a5d8b21c74: Status 404 returned error can't find the container with id 50fd4a837d1827c0ea6d7d471e707f8c43db11e9a48190defcb7c9a5d8b21c74
Jul 27 16:27:20.103044 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:20.102971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8" event={"ID":"2e603be2-8915-481b-9db0-4bf942836c39","Type":"ContainerStarted","Data":"50fd4a837d1827c0ea6d7d471e707f8c43db11e9a48190defcb7c9a5d8b21c74"}
Jul 27 16:27:21.108851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:21.108766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8" event={"ID":"2e603be2-8915-481b-9db0-4bf942836c39","Type":"ContainerStarted","Data":"1c546e3bbe9ad2d827f3096fdd2bc8c1bde0fba0ec930166edfa0d17bbb2e0d6"}
Jul 27 16:27:21.109207 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:21.108864    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:21.127367 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:21.127307    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8" podStartSLOduration=1.611479096 podStartE2EDuration="3.127263567s" totalImagesPullingTime="1.515784471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:21.126589646 +0000 UTC m=+641.027730096" watchObservedRunningTime="2026-07-27 16:27:21.127263567 +0000 UTC m=+641.028404020"
Jul 27 16:27:29.165877 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:29.165837    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:29.264990 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:29.264956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:32.116497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:32.116467    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-qxpr8"
Jul 27 16:27:39.122622 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:39.122582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:43.367859 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.367823    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"]
Jul 27 16:27:43.372157 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.372140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.376527 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.376502    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 16:27:43.376647 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.376507    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgshm\""
Jul 27 16:27:43.376647 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.376507    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 27 16:27:43.376647 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.376598    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 16:27:43.385197 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.385174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"]
Jul 27 16:27:43.402777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.402714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:43.432694 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.432664    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.432828 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.432699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.432828 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.432750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.432828 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.432802    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmzj\" (UniqueName: \"kubernetes.io/projected/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-kube-api-access-7vmzj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.432926 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.432830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.432926 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.432847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.533927 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.533894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.533927 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.533926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534161 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.533960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534161 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.533983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534161 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.534014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534161 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.534057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmzj\" (UniqueName: \"kubernetes.io/projected/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-kube-api-access-7vmzj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534388 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.534349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.534403    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.534518 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.534489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.536168 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.536143    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.536440 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.536423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.543347 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.543301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmzj\" (UniqueName: \"kubernetes.io/projected/4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7-kube-api-access-7vmzj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nv7n2\" (UID: \"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.682578 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.682546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:43.812929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.812899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"]
Jul 27 16:27:43.814523 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:27:43.814499    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f52ebfd_4a7b_49b2_b3f1_0160e46f04d7.slice/crio-3584e8898983ae3f5a174c73ac398c4b08986d29e5a5cbd2b265e769802c4720 WatchSource:0}: Error finding container 3584e8898983ae3f5a174c73ac398c4b08986d29e5a5cbd2b265e769802c4720: Status 404 returned error can't find the container with id 3584e8898983ae3f5a174c73ac398c4b08986d29e5a5cbd2b265e769802c4720
Jul 27 16:27:43.815792 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:43.815772    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:27:44.200886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:44.200839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2" event={"ID":"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7","Type":"ContainerStarted","Data":"3584e8898983ae3f5a174c73ac398c4b08986d29e5a5cbd2b265e769802c4720"}
Jul 27 16:27:45.030203 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:45.030112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:45.089119 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:45.089046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:45.206216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:45.206174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2" event={"ID":"4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7","Type":"ContainerStarted","Data":"6ad4ce3d43fcf0301247264af81bceafec71e2ce6f0bc9a300474d55e630f440"}
Jul 27 16:27:45.239211 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:45.239163    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2" podStartSLOduration=1.279296406 podStartE2EDuration="2.239148372s" totalImagesPullingTime="959.851966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:45.238003452 +0000 UTC m=+665.139143900" watchObservedRunningTime="2026-07-27 16:27:45.239148372 +0000 UTC m=+665.140288819"
Jul 27 16:27:46.965634 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:46.965604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:53.683243 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:53.683198    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:53.683809 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:53.683261    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:53.695787 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:53.695755    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:54.252555 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:54.252519    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nv7n2"
Jul 27 16:27:54.697112 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:54.697083    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:54.767962 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:54.767917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:27:58.671946 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:27:58.671905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:28:06.975158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:06.975127    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"]
Jul 27 16:28:06.978208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:06.978190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:06.982131 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:06.982113    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 27 16:28:06.991211 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:06.991190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"]
Jul 27 16:28:07.002672 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.002646    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:28:07.028445 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.028420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.028582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.028459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.028582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.028537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.028711 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.028599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df344889-1bb9-4d2c-8699-c8580ef4ef25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.028711 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.028625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrt54\" (UniqueName: \"kubernetes.io/projected/df344889-1bb9-4d2c-8699-c8580ef4ef25-kube-api-access-wrt54\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.028711 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.028673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129446 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df344889-1bb9-4d2c-8699-c8580ef4ef25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrt54\" (UniqueName: \"kubernetes.io/projected/df344889-1bb9-4d2c-8699-c8580ef4ef25-kube-api-access-wrt54\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129600    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129644 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.129904 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.130004 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.130064 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.129989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.131967 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.131939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df344889-1bb9-4d2c-8699-c8580ef4ef25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.132410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.132388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df344889-1bb9-4d2c-8699-c8580ef4ef25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.142208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.142178    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrt54\" (UniqueName: \"kubernetes.io/projected/df344889-1bb9-4d2c-8699-c8580ef4ef25-kube-api-access-wrt54\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4\" (UID: \"df344889-1bb9-4d2c-8699-c8580ef4ef25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.287572 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.287484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:07.415187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:07.415159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"]
Jul 27 16:28:07.417559 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:28:07.417528    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf344889_1bb9_4d2c_8699_c8580ef4ef25.slice/crio-47b33dcf9dc38073b90487a4b2c45ce293949f562aac4a7f667a14bdf5a88a76 WatchSource:0}: Error finding container 47b33dcf9dc38073b90487a4b2c45ce293949f562aac4a7f667a14bdf5a88a76: Status 404 returned error can't find the container with id 47b33dcf9dc38073b90487a4b2c45ce293949f562aac4a7f667a14bdf5a88a76
Jul 27 16:28:08.299461 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:08.299421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4" event={"ID":"df344889-1bb9-4d2c-8699-c8580ef4ef25","Type":"ContainerStarted","Data":"5b8a7b9ac18b2e45e37e0e3de1b89a55c8008fae5c5af1328b5c32d55768fa74"}
Jul 27 16:28:08.299882 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:08.299499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4" event={"ID":"df344889-1bb9-4d2c-8699-c8580ef4ef25","Type":"ContainerStarted","Data":"47b33dcf9dc38073b90487a4b2c45ce293949f562aac4a7f667a14bdf5a88a76"}
Jul 27 16:28:08.333489 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:08.333443    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4" podStartSLOduration=2.114238108 podStartE2EDuration="2.333427941s" totalImagesPullingTime="219.189833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:28:08.33267605 +0000 UTC m=+688.233816499" watchObservedRunningTime="2026-07-27 16:28:08.333427941 +0000 UTC m=+688.234568388"
Jul 27 16:28:17.287965 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:17.287930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:17.288365 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:17.287990    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:17.300601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:17.300578    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:17.344900 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:17.344876    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4"
Jul 27 16:28:27.254675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:27.254636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:28:48.799711 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:28:48.799614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:29:44.194585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:29:44.194546    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:29:45.805007 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:29:45.804973    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:30:00.156584 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.156546    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"]
Jul 27 16:30:00.159722 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.159703    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"
Jul 27 16:30:00.162604 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.162583    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\""
Jul 27 16:30:00.169151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.169039    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"
Jul 27 16:30:00.171520 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.171494    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"]
Jul 27 16:30:00.304579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.304538    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"]
Jul 27 16:30:00.306994 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:30:00.306960    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370f01cc_ac79_4033_ac93_bde8166b53dc.slice/crio-3962e81edb2a2be5f00d251b53c81ad893b3607b3d066cef17904ffb5efdfcb5 WatchSource:0}: Error finding container 3962e81edb2a2be5f00d251b53c81ad893b3607b3d066cef17904ffb5efdfcb5: Status 404 returned error can't find the container with id 3962e81edb2a2be5f00d251b53c81ad893b3607b3d066cef17904ffb5efdfcb5
Jul 27 16:30:00.745496 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:00.745456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5" event={"ID":"370f01cc-ac79-4033-ac93-bde8166b53dc","Type":"ContainerStarted","Data":"3962e81edb2a2be5f00d251b53c81ad893b3607b3d066cef17904ffb5efdfcb5"}
Jul 27 16:30:03.761803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:03.761700    2559 generic.go:356] "Generic (PLEG): container finished" podID="370f01cc-ac79-4033-ac93-bde8166b53dc" containerID="b03fe7517104ccb0861a793ff288523a014326a515c83b60ff5829cabe3dfc80" exitCode=0
Jul 27 16:30:03.761803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:03.761786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5" event={"ID":"370f01cc-ac79-4033-ac93-bde8166b53dc","Type":"ContainerDied","Data":"b03fe7517104ccb0861a793ff288523a014326a515c83b60ff5829cabe3dfc80"}
Jul 27 16:30:04.890811 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:04.890785    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"
Jul 27 16:30:05.771045 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:05.771013    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5"
Jul 27 16:30:05.771045 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:05.771023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-rhkb5" event={"ID":"370f01cc-ac79-4033-ac93-bde8166b53dc","Type":"ContainerDied","Data":"3962e81edb2a2be5f00d251b53c81ad893b3607b3d066cef17904ffb5efdfcb5"}
Jul 27 16:30:05.771347 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:05.771072    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3962e81edb2a2be5f00d251b53c81ad893b3607b3d066cef17904ffb5efdfcb5"
Jul 27 16:30:35.881595 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:35.881561    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-556f76c959-5r57l"]
Jul 27 16:30:35.882113 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:35.881799    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-556f76c959-5r57l" podUID="92eba6f2-c76e-45a1-80d1-6fc2b0842b83" containerName="manager" containerID="cri-o://77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c" gracePeriod=30
Jul 27 16:30:36.128784 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.128756    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:30:36.228086 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.228051    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-webhook-cert\") pod \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") "
Jul 27 16:30:36.228314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.228129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7598\" (UniqueName: \"kubernetes.io/projected/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-kube-api-access-t7598\") pod \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\" (UID: \"92eba6f2-c76e-45a1-80d1-6fc2b0842b83\") "
Jul 27 16:30:36.230149 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.230107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "92eba6f2-c76e-45a1-80d1-6fc2b0842b83" (UID: "92eba6f2-c76e-45a1-80d1-6fc2b0842b83"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:30:36.230265 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.230223    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-kube-api-access-t7598" (OuterVolumeSpecName: "kube-api-access-t7598") pod "92eba6f2-c76e-45a1-80d1-6fc2b0842b83" (UID: "92eba6f2-c76e-45a1-80d1-6fc2b0842b83"). InnerVolumeSpecName "kube-api-access-t7598". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:30:36.328751 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.328697    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7598\" (UniqueName: \"kubernetes.io/projected/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-kube-api-access-t7598\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:30:36.328751 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.328744    2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92eba6f2-c76e-45a1-80d1-6fc2b0842b83-webhook-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:30:36.898786 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.898750    2559 generic.go:356] "Generic (PLEG): container finished" podID="92eba6f2-c76e-45a1-80d1-6fc2b0842b83" containerID="77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c" exitCode=0
Jul 27 16:30:36.899196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.898797    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-5r57l"
Jul 27 16:30:36.899196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.898835    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-5r57l" event={"ID":"92eba6f2-c76e-45a1-80d1-6fc2b0842b83","Type":"ContainerDied","Data":"77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c"}
Jul 27 16:30:36.899196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.898889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-5r57l" event={"ID":"92eba6f2-c76e-45a1-80d1-6fc2b0842b83","Type":"ContainerDied","Data":"c8559a82ef4eba87193132bdc6bba3b717359f66df48c14ebfde0e8000efda87"}
Jul 27 16:30:36.899196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.898919    2559 scope.go:117] "RemoveContainer" containerID="77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c"
Jul 27 16:30:36.907121 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.907100    2559 scope.go:117] "RemoveContainer" containerID="77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c"
Jul 27 16:30:36.907428 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:30:36.907410    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c\": container with ID starting with 77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c not found: ID does not exist" containerID="77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c"
Jul 27 16:30:36.907483 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.907445    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c"} err="failed to get container status \"77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c\": rpc error: code = NotFound desc = could not find container \"77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c\": container with ID starting with 77d63214b56ffd22399debeeb94be86507bccb023d62443e10bfcce566bb520c not found: ID does not exist"
Jul 27 16:30:36.918387 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.918154    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-556f76c959-5r57l"]
Jul 27 16:30:36.920780 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:36.920754    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-556f76c959-5r57l"]
Jul 27 16:30:38.736228 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:30:38.736195    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92eba6f2-c76e-45a1-80d1-6fc2b0842b83" path="/var/lib/kubelet/pods/92eba6f2-c76e-45a1-80d1-6fc2b0842b83/volumes"
Jul 27 16:31:37.753557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753453    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-556f76c959-dtjjr"]
Jul 27 16:31:37.754380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753736    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92eba6f2-c76e-45a1-80d1-6fc2b0842b83" containerName="manager"
Jul 27 16:31:37.754380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753746    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="92eba6f2-c76e-45a1-80d1-6fc2b0842b83" containerName="manager"
Jul 27 16:31:37.754380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753754    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370f01cc-ac79-4033-ac93-bde8166b53dc" containerName="cleanup"
Jul 27 16:31:37.754380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753760    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="370f01cc-ac79-4033-ac93-bde8166b53dc" containerName="cleanup"
Jul 27 16:31:37.754380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753830    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="370f01cc-ac79-4033-ac93-bde8166b53dc" containerName="cleanup"
Jul 27 16:31:37.754380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.753839    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="92eba6f2-c76e-45a1-80d1-6fc2b0842b83" containerName="manager"
Jul 27 16:31:37.756756 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.756736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:37.760559 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.760534    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wn5rq\""
Jul 27 16:31:37.760559 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.760554    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 16:31:37.760699 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.760534    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 16:31:37.769195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.769169    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556f76c959-dtjjr"]
Jul 27 16:31:37.824818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.824794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/056fb472-a2df-4156-b538-b37f4b4bf399-webhook-cert\") pod \"maas-controller-556f76c959-dtjjr\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") " pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:37.824910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.824837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q57\" (UniqueName: \"kubernetes.io/projected/056fb472-a2df-4156-b538-b37f4b4bf399-kube-api-access-s6q57\") pod \"maas-controller-556f76c959-dtjjr\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") " pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:37.926027 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.925986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/056fb472-a2df-4156-b538-b37f4b4bf399-webhook-cert\") pod \"maas-controller-556f76c959-dtjjr\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") " pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:37.926225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.926042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q57\" (UniqueName: \"kubernetes.io/projected/056fb472-a2df-4156-b538-b37f4b4bf399-kube-api-access-s6q57\") pod \"maas-controller-556f76c959-dtjjr\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") " pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:37.928549 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.928527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/056fb472-a2df-4156-b538-b37f4b4bf399-webhook-cert\") pod \"maas-controller-556f76c959-dtjjr\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") " pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:37.935309 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:37.935256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q57\" (UniqueName: \"kubernetes.io/projected/056fb472-a2df-4156-b538-b37f4b4bf399-kube-api-access-s6q57\") pod \"maas-controller-556f76c959-dtjjr\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") " pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:38.082074 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:38.081980    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:38.214087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:38.214062    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556f76c959-dtjjr"]
Jul 27 16:31:38.216155 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:31:38.216128    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056fb472_a2df_4156_b538_b37f4b4bf399.slice/crio-4923c7f5d37c2b5c0b54c2511b6f395ee45e2350b6a7c707d75bc4ec2a05067e WatchSource:0}: Error finding container 4923c7f5d37c2b5c0b54c2511b6f395ee45e2350b6a7c707d75bc4ec2a05067e: Status 404 returned error can't find the container with id 4923c7f5d37c2b5c0b54c2511b6f395ee45e2350b6a7c707d75bc4ec2a05067e
Jul 27 16:31:39.155263 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:39.155165    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-dtjjr" event={"ID":"056fb472-a2df-4156-b538-b37f4b4bf399","Type":"ContainerStarted","Data":"650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71"}
Jul 27 16:31:39.155263 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:39.155216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-dtjjr" event={"ID":"056fb472-a2df-4156-b538-b37f4b4bf399","Type":"ContainerStarted","Data":"4923c7f5d37c2b5c0b54c2511b6f395ee45e2350b6a7c707d75bc4ec2a05067e"}
Jul 27 16:31:39.155263 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:39.155247    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:39.177450 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:39.177373    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-556f76c959-dtjjr" podStartSLOduration=1.497427826 podStartE2EDuration="2.177358084s" totalImagesPullingTime="679.930258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:31:39.176409186 +0000 UTC m=+899.077549633" watchObservedRunningTime="2026-07-27 16:31:39.177358084 +0000 UTC m=+899.078498532"
Jul 27 16:31:50.161976 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:50.161943    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:54.758645 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:54.758596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:31:54.822599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:54.822564    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-556f76c959-dtjjr"]
Jul 27 16:31:54.822789 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:54.822761    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-556f76c959-dtjjr" podUID="056fb472-a2df-4156-b538-b37f4b4bf399" containerName="manager" containerID="cri-o://650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71" gracePeriod=30
Jul 27 16:31:55.069685 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.069656    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:55.177714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.177671    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/056fb472-a2df-4156-b538-b37f4b4bf399-webhook-cert\") pod \"056fb472-a2df-4156-b538-b37f4b4bf399\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") "
Jul 27 16:31:55.177882 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.177822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6q57\" (UniqueName: \"kubernetes.io/projected/056fb472-a2df-4156-b538-b37f4b4bf399-kube-api-access-s6q57\") pod \"056fb472-a2df-4156-b538-b37f4b4bf399\" (UID: \"056fb472-a2df-4156-b538-b37f4b4bf399\") "
Jul 27 16:31:55.179781 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.179755    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056fb472-a2df-4156-b538-b37f4b4bf399-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "056fb472-a2df-4156-b538-b37f4b4bf399" (UID: "056fb472-a2df-4156-b538-b37f4b4bf399"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:31:55.179844 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.179800    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056fb472-a2df-4156-b538-b37f4b4bf399-kube-api-access-s6q57" (OuterVolumeSpecName: "kube-api-access-s6q57") pod "056fb472-a2df-4156-b538-b37f4b4bf399" (UID: "056fb472-a2df-4156-b538-b37f4b4bf399"). InnerVolumeSpecName "kube-api-access-s6q57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:31:55.224493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.224458    2559 generic.go:356] "Generic (PLEG): container finished" podID="056fb472-a2df-4156-b538-b37f4b4bf399" containerID="650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71" exitCode=0
Jul 27 16:31:55.224665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.224507    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-dtjjr"
Jul 27 16:31:55.224665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.224550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-dtjjr" event={"ID":"056fb472-a2df-4156-b538-b37f4b4bf399","Type":"ContainerDied","Data":"650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71"}
Jul 27 16:31:55.224665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.224634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-dtjjr" event={"ID":"056fb472-a2df-4156-b538-b37f4b4bf399","Type":"ContainerDied","Data":"4923c7f5d37c2b5c0b54c2511b6f395ee45e2350b6a7c707d75bc4ec2a05067e"}
Jul 27 16:31:55.224665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.224658    2559 scope.go:117] "RemoveContainer" containerID="650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71"
Jul 27 16:31:55.233044 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.233023    2559 scope.go:117] "RemoveContainer" containerID="650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71"
Jul 27 16:31:55.233328 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:31:55.233307    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71\": container with ID starting with 650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71 not found: ID does not exist" containerID="650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71"
Jul 27 16:31:55.233403 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.233340    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71"} err="failed to get container status \"650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71\": rpc error: code = NotFound desc = could not find container \"650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71\": container with ID starting with 650415af35eb8ed94ed4415b1d9f8aab8f6059396d08d02926481fe777134c71 not found: ID does not exist"
Jul 27 16:31:55.247581 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.247552    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-556f76c959-dtjjr"]
Jul 27 16:31:55.250631 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.250603    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-556f76c959-dtjjr"]
Jul 27 16:31:55.279334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.279306    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6q57\" (UniqueName: \"kubernetes.io/projected/056fb472-a2df-4156-b538-b37f4b4bf399-kube-api-access-s6q57\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:31:55.279334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.279333    2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/056fb472-a2df-4156-b538-b37f4b4bf399-webhook-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:31:55.931058 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:55.931018    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:31:56.033717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.033686    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-556f76c959-zgl78"]
Jul 27 16:31:56.033987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.033975    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056fb472-a2df-4156-b538-b37f4b4bf399" containerName="manager"
Jul 27 16:31:56.034031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.033989    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="056fb472-a2df-4156-b538-b37f4b4bf399" containerName="manager"
Jul 27 16:31:56.034070 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.034045    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="056fb472-a2df-4156-b538-b37f4b4bf399" containerName="manager"
Jul 27 16:31:56.038075 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.038058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.041060 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.041039    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 16:31:56.042373 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.042359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 16:31:56.042438 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.042397    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wn5rq\""
Jul 27 16:31:56.051457 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.051432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556f76c959-zgl78"]
Jul 27 16:31:56.187512 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.187403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e69829-dc4a-46b8-9a5c-35cc02541ab9-webhook-cert\") pod \"maas-controller-556f76c959-zgl78\" (UID: \"e3e69829-dc4a-46b8-9a5c-35cc02541ab9\") " pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.187512 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.187490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxhp\" (UniqueName: \"kubernetes.io/projected/e3e69829-dc4a-46b8-9a5c-35cc02541ab9-kube-api-access-jrxhp\") pod \"maas-controller-556f76c959-zgl78\" (UID: \"e3e69829-dc4a-46b8-9a5c-35cc02541ab9\") " pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.288573 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.288520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxhp\" (UniqueName: \"kubernetes.io/projected/e3e69829-dc4a-46b8-9a5c-35cc02541ab9-kube-api-access-jrxhp\") pod \"maas-controller-556f76c959-zgl78\" (UID: \"e3e69829-dc4a-46b8-9a5c-35cc02541ab9\") " pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.288730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.288601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e69829-dc4a-46b8-9a5c-35cc02541ab9-webhook-cert\") pod \"maas-controller-556f76c959-zgl78\" (UID: \"e3e69829-dc4a-46b8-9a5c-35cc02541ab9\") " pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.291023 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.290993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3e69829-dc4a-46b8-9a5c-35cc02541ab9-webhook-cert\") pod \"maas-controller-556f76c959-zgl78\" (UID: \"e3e69829-dc4a-46b8-9a5c-35cc02541ab9\") " pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.301896 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.301866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxhp\" (UniqueName: \"kubernetes.io/projected/e3e69829-dc4a-46b8-9a5c-35cc02541ab9-kube-api-access-jrxhp\") pod \"maas-controller-556f76c959-zgl78\" (UID: \"e3e69829-dc4a-46b8-9a5c-35cc02541ab9\") " pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.347535 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.347492    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:56.482134 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.482107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-556f76c959-zgl78"]
Jul 27 16:31:56.484213 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:31:56.484186    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e69829_dc4a_46b8_9a5c_35cc02541ab9.slice/crio-3383c6e43d975034bbb6b8c103adf0c814b003f55192f6b65e1eb097c30a054f WatchSource:0}: Error finding container 3383c6e43d975034bbb6b8c103adf0c814b003f55192f6b65e1eb097c30a054f: Status 404 returned error can't find the container with id 3383c6e43d975034bbb6b8c103adf0c814b003f55192f6b65e1eb097c30a054f
Jul 27 16:31:56.735185 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:56.735106    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056fb472-a2df-4156-b538-b37f4b4bf399" path="/var/lib/kubelet/pods/056fb472-a2df-4156-b538-b37f4b4bf399/volumes"
Jul 27 16:31:57.234593 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:57.234556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-zgl78" event={"ID":"e3e69829-dc4a-46b8-9a5c-35cc02541ab9","Type":"ContainerStarted","Data":"7a8c2168976c1c7da96fb6a6dbf47d92a1a7a377f0b10981133fd60699ba3be9"}
Jul 27 16:31:57.235026 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:57.234616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-556f76c959-zgl78" event={"ID":"e3e69829-dc4a-46b8-9a5c-35cc02541ab9","Type":"ContainerStarted","Data":"3383c6e43d975034bbb6b8c103adf0c814b003f55192f6b65e1eb097c30a054f"}
Jul 27 16:31:57.235026 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:57.234701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:31:57.255871 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:31:57.255766    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-556f76c959-zgl78" podStartSLOduration=0.751980147 podStartE2EDuration="1.255749482s" totalImagesPullingTime="503.769335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:31:57.254293739 +0000 UTC m=+917.155434183" watchObservedRunningTime="2026-07-27 16:31:57.255749482 +0000 UTC m=+917.156889929"
Jul 27 16:32:01.447461 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:01.447423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:08.241762 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:08.241731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-556f76c959-zgl78"
Jul 27 16:32:28.754864 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:28.754826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:29.604218 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:29.604181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:29.703129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:29.703093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:29.823117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:29.823083    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:35.798204 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:35.798160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:35.860732 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:35.860697    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:36.051781 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:36.051699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:36.154601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:36.154565    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:36.257140 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:36.257109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:42.482028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:42.481993    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:32:42.583749 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:32:42.583710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:34:43.391573 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:34:43.391495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:37:29.458055 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:37:29.458012    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:37:31.200339 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:37:31.200303    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:39:57.855569 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:39:57.855534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:40:26.589060 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:40:26.589019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:40:52.756087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:40:52.756055    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:41:02.951105 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:41:02.951067    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:41:55.386778 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:41:55.386735    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:41:56.896398 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:41:56.896356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:43:43.030683 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:43:43.030652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:43:45.056275 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:43:45.056242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:00.143547 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.143506    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"]
Jul 27 16:45:00.146883 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.146865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"
Jul 27 16:45:00.149583 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.149561    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\""
Jul 27 16:45:00.156138 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.156117    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"
Jul 27 16:45:00.199745 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.199714    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"]
Jul 27 16:45:00.282023 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.281989    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"]
Jul 27 16:45:00.285249 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:45:00.285223    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc5fbda_d461_4376_a9f3_df29107724b2.slice/crio-744052f017cfd1b0b95bd72a52ff5474b7dcdd8ece4169c3c02d64855adb0794 WatchSource:0}: Error finding container 744052f017cfd1b0b95bd72a52ff5474b7dcdd8ece4169c3c02d64855adb0794: Status 404 returned error can't find the container with id 744052f017cfd1b0b95bd72a52ff5474b7dcdd8ece4169c3c02d64855adb0794
Jul 27 16:45:00.395564 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:00.395479    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq" event={"ID":"1cc5fbda-d461-4376-a9f3-df29107724b2","Type":"ContainerStarted","Data":"744052f017cfd1b0b95bd72a52ff5474b7dcdd8ece4169c3c02d64855adb0794"}
Jul 27 16:45:01.400876 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:01.400781    2559 generic.go:356] "Generic (PLEG): container finished" podID="1cc5fbda-d461-4376-a9f3-df29107724b2" containerID="c438a4f0e02bc8cabced39d4102e27baccf00a56a6b04f8911b548faa8440660" exitCode=0
Jul 27 16:45:01.401209 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:01.400875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq" event={"ID":"1cc5fbda-d461-4376-a9f3-df29107724b2","Type":"ContainerDied","Data":"c438a4f0e02bc8cabced39d4102e27baccf00a56a6b04f8911b548faa8440660"}
Jul 27 16:45:02.530577 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:02.530549    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"
Jul 27 16:45:03.409742 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:03.409699    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq" event={"ID":"1cc5fbda-d461-4376-a9f3-df29107724b2","Type":"ContainerDied","Data":"744052f017cfd1b0b95bd72a52ff5474b7dcdd8ece4169c3c02d64855adb0794"}
Jul 27 16:45:03.409742 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:03.409725    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-jpbwq"
Jul 27 16:45:03.409949 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:03.409799    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744052f017cfd1b0b95bd72a52ff5474b7dcdd8ece4169c3c02d64855adb0794"
Jul 27 16:45:08.861187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:08.861150    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:34.593729 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:34.593694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:37.616438 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:37.616379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:37.713336 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:37.713266    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:38.124113 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:38.124077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:44.092717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:44.092685    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:44.155271 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:44.155238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:44.319408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:44.319360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:44.622660 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:44.622603    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:44.926723 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:44.926694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:47.268726 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.268656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:47.349032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.348991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:47.383728 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.383696    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"]
Jul 27 16:45:47.384016 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.384005    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc5fbda-d461-4376-a9f3-df29107724b2" containerName="cleanup"
Jul 27 16:45:47.384058 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.384018    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc5fbda-d461-4376-a9f3-df29107724b2" containerName="cleanup"
Jul 27 16:45:47.384090 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.384069    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc5fbda-d461-4376-a9f3-df29107724b2" containerName="cleanup"
Jul 27 16:45:47.386773 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.386757    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.390836 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.390817    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:45:47.390978 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.390962    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:45:47.391956 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.391937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a178d363-serving-cert\""
Jul 27 16:45:47.392039 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.391946    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:45:47.399922 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.399898    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"]
Jul 27 16:45:47.434316 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.434252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:47.540822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.540727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ca07-cfa6-425b-a30b-36531843177f-openshift-service-ca\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.540822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.540773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a184ca07-cfa6-425b-a30b-36531843177f-maas-api-tls\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.541003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.540864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtplm\" (UniqueName: \"kubernetes.io/projected/a184ca07-cfa6-425b-a30b-36531843177f-kube-api-access-vtplm\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.642154 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.642119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a184ca07-cfa6-425b-a30b-36531843177f-maas-api-tls\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.642366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.642172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtplm\" (UniqueName: \"kubernetes.io/projected/a184ca07-cfa6-425b-a30b-36531843177f-kube-api-access-vtplm\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.642366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.642214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ca07-cfa6-425b-a30b-36531843177f-openshift-service-ca\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.642806 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.642785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ca07-cfa6-425b-a30b-36531843177f-openshift-service-ca\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.644606 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.644583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a184ca07-cfa6-425b-a30b-36531843177f-maas-api-tls\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.658109 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.658083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtplm\" (UniqueName: \"kubernetes.io/projected/a184ca07-cfa6-425b-a30b-36531843177f-kube-api-access-vtplm\") pod \"maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.697176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.697142    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:47.701943 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.701903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:47.832131 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.832101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"]
Jul 27 16:45:47.834647 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:45:47.834621    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda184ca07_cfa6_425b_a30b_36531843177f.slice/crio-db728e644f97edcc076cbcb2e3929cfde815cd340653239115d3f8fe68356cd0 WatchSource:0}: Error finding container db728e644f97edcc076cbcb2e3929cfde815cd340653239115d3f8fe68356cd0: Status 404 returned error can't find the container with id db728e644f97edcc076cbcb2e3929cfde815cd340653239115d3f8fe68356cd0
Jul 27 16:45:47.835990 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:47.835973    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:45:48.199348 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.199315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:48.306561 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.306522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:45:48.323872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.323838    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"]
Jul 27 16:45:48.328062 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.328043    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.330884 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.330862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a178d363\""
Jul 27 16:45:48.339122 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.339097    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"]
Jul 27 16:45:48.340735 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.340714    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"]
Jul 27 16:45:48.343650 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.343635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.346454 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.346439    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a178d363-dockercfg-ngpcr\""
Jul 27 16:45:48.355101 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.355075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"]
Jul 27 16:45:48.448035 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.448003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa621ff-a57b-4263-86d8-9afa702b96d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.448208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.448054    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkk6k\" (UniqueName: \"kubernetes.io/projected/23bb59fb-8420-4576-b013-c408dbc5b04e-kube-api-access-qkk6k\") pod \"payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") " pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.448208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.448073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxd4d\" (UniqueName: \"kubernetes.io/projected/7aa621ff-a57b-4263-86d8-9afa702b96d4-kube-api-access-qxd4d\") pod \"payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.448208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.448172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb59fb-8420-4576-b013-c408dbc5b04e-plugins-config-volume\") pod \"payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") " pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.549396 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.549308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa621ff-a57b-4263-86d8-9afa702b96d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.549396 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.549363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkk6k\" (UniqueName: \"kubernetes.io/projected/23bb59fb-8420-4576-b013-c408dbc5b04e-kube-api-access-qkk6k\") pod \"payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") " pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.549578 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.549481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxd4d\" (UniqueName: \"kubernetes.io/projected/7aa621ff-a57b-4263-86d8-9afa702b96d4-kube-api-access-qxd4d\") pod \"payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.549578 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.549546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb59fb-8420-4576-b013-c408dbc5b04e-plugins-config-volume\") pod \"payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") " pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.550127 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.550108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa621ff-a57b-4263-86d8-9afa702b96d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.550181 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.550165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb59fb-8420-4576-b013-c408dbc5b04e-plugins-config-volume\") pod \"payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") " pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.559194 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.559166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkk6k\" (UniqueName: \"kubernetes.io/projected/23bb59fb-8420-4576-b013-c408dbc5b04e-kube-api-access-qkk6k\") pod \"payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") " pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.560210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.560190    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxd4d\" (UniqueName: \"kubernetes.io/projected/7aa621ff-a57b-4263-86d8-9afa702b96d4-kube-api-access-qxd4d\") pod \"payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.586557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.586524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n" event={"ID":"a184ca07-cfa6-425b-a30b-36531843177f","Type":"ContainerStarted","Data":"db728e644f97edcc076cbcb2e3929cfde815cd340653239115d3f8fe68356cd0"}
Jul 27 16:45:48.639404 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.639367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:48.653588 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.653558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:48.786698 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.786660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"]
Jul 27 16:45:48.790203 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:45:48.790173    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa621ff_a57b_4263_86d8_9afa702b96d4.slice/crio-e7ca7502402574d59314b3a9894ee2fc1d3d17f41e0ceae69fc7f3cbd9255f24 WatchSource:0}: Error finding container e7ca7502402574d59314b3a9894ee2fc1d3d17f41e0ceae69fc7f3cbd9255f24: Status 404 returned error can't find the container with id e7ca7502402574d59314b3a9894ee2fc1d3d17f41e0ceae69fc7f3cbd9255f24
Jul 27 16:45:48.804130 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:48.804049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"]
Jul 27 16:45:48.808086 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:45:48.808060    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bb59fb_8420_4576_b013_c408dbc5b04e.slice/crio-76815f0f50f177f9cf7429deda471839bcb7f1d8abda1ce69bc13a1da9f42b7e WatchSource:0}: Error finding container 76815f0f50f177f9cf7429deda471839bcb7f1d8abda1ce69bc13a1da9f42b7e: Status 404 returned error can't find the container with id 76815f0f50f177f9cf7429deda471839bcb7f1d8abda1ce69bc13a1da9f42b7e
Jul 27 16:45:49.591983 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.591947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj" event={"ID":"7aa621ff-a57b-4263-86d8-9afa702b96d4","Type":"ContainerStarted","Data":"d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56"}
Jul 27 16:45:49.592440 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.592027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj" event={"ID":"7aa621ff-a57b-4263-86d8-9afa702b96d4","Type":"ContainerStarted","Data":"e7ca7502402574d59314b3a9894ee2fc1d3d17f41e0ceae69fc7f3cbd9255f24"}
Jul 27 16:45:49.592440 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.592053    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:45:49.593550 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.593520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7" event={"ID":"23bb59fb-8420-4576-b013-c408dbc5b04e","Type":"ContainerStarted","Data":"b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c"}
Jul 27 16:45:49.593646 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.593568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7" event={"ID":"23bb59fb-8420-4576-b013-c408dbc5b04e","Type":"ContainerStarted","Data":"76815f0f50f177f9cf7429deda471839bcb7f1d8abda1ce69bc13a1da9f42b7e"}
Jul 27 16:45:49.593646 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.593610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:45:49.611554 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.611497    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj" podStartSLOduration=1.6114830549999999 podStartE2EDuration="1.611483055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:49.60968798 +0000 UTC m=+1749.510828430" watchObservedRunningTime="2026-07-27 16:45:49.611483055 +0000 UTC m=+1749.512623502"
Jul 27 16:45:49.627687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:49.627637    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7" podStartSLOduration=1.627624562 podStartE2EDuration="1.627624562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:49.627090265 +0000 UTC m=+1749.528230712" watchObservedRunningTime="2026-07-27 16:45:49.627624562 +0000 UTC m=+1749.528765008"
Jul 27 16:45:50.598936 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:50.598888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n" event={"ID":"a184ca07-cfa6-425b-a30b-36531843177f","Type":"ContainerStarted","Data":"886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d"}
Jul 27 16:45:50.622553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:50.622503    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n" podStartSLOduration=1.377555184 podStartE2EDuration="3.622487274s" totalImagesPullingTime="2.24493209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:50.621052345 +0000 UTC m=+1750.522192790" watchObservedRunningTime="2026-07-27 16:45:50.622487274 +0000 UTC m=+1750.523627720"
Jul 27 16:45:51.603323 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:51.603295    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:45:57.612216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:45:57.612189    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:46:00.601450 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:00.601421    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:46:00.601852 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:00.601577    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:46:23.257631 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.257595    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"]
Jul 27 16:46:23.258157 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.257852    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n" podUID="a184ca07-cfa6-425b-a30b-36531843177f" containerName="maas-api" containerID="cri-o://886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d" gracePeriod=30
Jul 27 16:46:23.283621 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.283590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:23.312522 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.312483    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"]
Jul 27 16:46:23.312773 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.312744    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7" podUID="23bb59fb-8420-4576-b013-c408dbc5b04e" containerName="payload-processing" containerID="cri-o://b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c" gracePeriod=30
Jul 27 16:46:23.317266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.317232    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"]
Jul 27 16:46:23.317551 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.317526    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj" podUID="7aa621ff-a57b-4263-86d8-9afa702b96d4" containerName="payload-pre-processing" containerID="cri-o://d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56" gracePeriod=30
Jul 27 16:46:23.361407 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.361375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:23.462412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.462379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:23.568401 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.568376    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:46:23.580159 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.580137    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:46:23.598036 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.598014    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:46:23.643266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643227    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxd4d\" (UniqueName: \"kubernetes.io/projected/7aa621ff-a57b-4263-86d8-9afa702b96d4-kube-api-access-qxd4d\") pod \"7aa621ff-a57b-4263-86d8-9afa702b96d4\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") "
Jul 27 16:46:23.643266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643270    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ca07-cfa6-425b-a30b-36531843177f-openshift-service-ca\") pod \"a184ca07-cfa6-425b-a30b-36531843177f\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") "
Jul 27 16:46:23.643517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643312    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a184ca07-cfa6-425b-a30b-36531843177f-maas-api-tls\") pod \"a184ca07-cfa6-425b-a30b-36531843177f\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") "
Jul 27 16:46:23.643517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643483    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb59fb-8420-4576-b013-c408dbc5b04e-plugins-config-volume\") pod \"23bb59fb-8420-4576-b013-c408dbc5b04e\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") "
Jul 27 16:46:23.643625 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643537    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtplm\" (UniqueName: \"kubernetes.io/projected/a184ca07-cfa6-425b-a30b-36531843177f-kube-api-access-vtplm\") pod \"a184ca07-cfa6-425b-a30b-36531843177f\" (UID: \"a184ca07-cfa6-425b-a30b-36531843177f\") "
Jul 27 16:46:23.643625 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643567    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkk6k\" (UniqueName: \"kubernetes.io/projected/23bb59fb-8420-4576-b013-c408dbc5b04e-kube-api-access-qkk6k\") pod \"23bb59fb-8420-4576-b013-c408dbc5b04e\" (UID: \"23bb59fb-8420-4576-b013-c408dbc5b04e\") "
Jul 27 16:46:23.643728 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643629    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa621ff-a57b-4263-86d8-9afa702b96d4-plugins-config-volume\") pod \"7aa621ff-a57b-4263-86d8-9afa702b96d4\" (UID: \"7aa621ff-a57b-4263-86d8-9afa702b96d4\") "
Jul 27 16:46:23.643728 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643646    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a184ca07-cfa6-425b-a30b-36531843177f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a184ca07-cfa6-425b-a30b-36531843177f" (UID: "a184ca07-cfa6-425b-a30b-36531843177f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:46:23.643965 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.643938    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23bb59fb-8420-4576-b013-c408dbc5b04e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23bb59fb-8420-4576-b013-c408dbc5b04e" (UID: "23bb59fb-8420-4576-b013-c408dbc5b04e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:46:23.644438 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.644413    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23bb59fb-8420-4576-b013-c408dbc5b04e-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.644557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.644441    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ca07-cfa6-425b-a30b-36531843177f-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.644557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.644447    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa621ff-a57b-4263-86d8-9afa702b96d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7aa621ff-a57b-4263-86d8-9afa702b96d4" (UID: "7aa621ff-a57b-4263-86d8-9afa702b96d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:46:23.645712 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.645683    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a184ca07-cfa6-425b-a30b-36531843177f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a184ca07-cfa6-425b-a30b-36531843177f" (UID: "a184ca07-cfa6-425b-a30b-36531843177f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:46:23.646093 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.646066    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a184ca07-cfa6-425b-a30b-36531843177f-kube-api-access-vtplm" (OuterVolumeSpecName: "kube-api-access-vtplm") pod "a184ca07-cfa6-425b-a30b-36531843177f" (UID: "a184ca07-cfa6-425b-a30b-36531843177f"). InnerVolumeSpecName "kube-api-access-vtplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:46:23.646181 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.646117    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa621ff-a57b-4263-86d8-9afa702b96d4-kube-api-access-qxd4d" (OuterVolumeSpecName: "kube-api-access-qxd4d") pod "7aa621ff-a57b-4263-86d8-9afa702b96d4" (UID: "7aa621ff-a57b-4263-86d8-9afa702b96d4"). InnerVolumeSpecName "kube-api-access-qxd4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:46:23.646437 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.646421    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bb59fb-8420-4576-b013-c408dbc5b04e-kube-api-access-qkk6k" (OuterVolumeSpecName: "kube-api-access-qkk6k") pod "23bb59fb-8420-4576-b013-c408dbc5b04e" (UID: "23bb59fb-8420-4576-b013-c408dbc5b04e"). InnerVolumeSpecName "kube-api-access-qkk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:46:23.737610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.737575    2559 generic.go:356] "Generic (PLEG): container finished" podID="23bb59fb-8420-4576-b013-c408dbc5b04e" containerID="b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c" exitCode=0
Jul 27 16:46:23.737783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.737620    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"
Jul 27 16:46:23.737783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.737668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7" event={"ID":"23bb59fb-8420-4576-b013-c408dbc5b04e","Type":"ContainerDied","Data":"b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c"}
Jul 27 16:46:23.737783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.737716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7" event={"ID":"23bb59fb-8420-4576-b013-c408dbc5b04e","Type":"ContainerDied","Data":"76815f0f50f177f9cf7429deda471839bcb7f1d8abda1ce69bc13a1da9f42b7e"}
Jul 27 16:46:23.737783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.737743    2559 scope.go:117] "RemoveContainer" containerID="b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c"
Jul 27 16:46:23.738968 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.738928    2559 generic.go:356] "Generic (PLEG): container finished" podID="a184ca07-cfa6-425b-a30b-36531843177f" containerID="886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d" exitCode=0
Jul 27 16:46:23.738968 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.738958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n" event={"ID":"a184ca07-cfa6-425b-a30b-36531843177f","Type":"ContainerDied","Data":"886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d"}
Jul 27 16:46:23.739114 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.738978    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"
Jul 27 16:46:23.739114 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.739001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n" event={"ID":"a184ca07-cfa6-425b-a30b-36531843177f","Type":"ContainerDied","Data":"db728e644f97edcc076cbcb2e3929cfde815cd340653239115d3f8fe68356cd0"}
Jul 27 16:46:23.740517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.740493    2559 generic.go:356] "Generic (PLEG): container finished" podID="7aa621ff-a57b-4263-86d8-9afa702b96d4" containerID="d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56" exitCode=0
Jul 27 16:46:23.740634 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.740516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj" event={"ID":"7aa621ff-a57b-4263-86d8-9afa702b96d4","Type":"ContainerDied","Data":"d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56"}
Jul 27 16:46:23.740634 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.740549    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"
Jul 27 16:46:23.740634 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.740560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj" event={"ID":"7aa621ff-a57b-4263-86d8-9afa702b96d4","Type":"ContainerDied","Data":"e7ca7502402574d59314b3a9894ee2fc1d3d17f41e0ceae69fc7f3cbd9255f24"}
Jul 27 16:46:23.744951 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.744932    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aa621ff-a57b-4263-86d8-9afa702b96d4-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.745048 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.744954    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxd4d\" (UniqueName: \"kubernetes.io/projected/7aa621ff-a57b-4263-86d8-9afa702b96d4-kube-api-access-qxd4d\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.745048 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.744973    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a184ca07-cfa6-425b-a30b-36531843177f-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.745048 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.744985    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtplm\" (UniqueName: \"kubernetes.io/projected/a184ca07-cfa6-425b-a30b-36531843177f-kube-api-access-vtplm\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.745048 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.744996    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkk6k\" (UniqueName: \"kubernetes.io/projected/23bb59fb-8420-4576-b013-c408dbc5b04e-kube-api-access-qkk6k\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:46:23.748384 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.748256    2559 scope.go:117] "RemoveContainer" containerID="b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c"
Jul 27 16:46:23.748636 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:46:23.748612    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c\": container with ID starting with b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c not found: ID does not exist" containerID="b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c"
Jul 27 16:46:23.748718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.748647    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c"} err="failed to get container status \"b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c\": rpc error: code = NotFound desc = could not find container \"b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c\": container with ID starting with b6c5f728913bca3660d3769dd976bd117628781472778c4dfe35a225cb34c09c not found: ID does not exist"
Jul 27 16:46:23.748718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.748669    2559 scope.go:117] "RemoveContainer" containerID="886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d"
Jul 27 16:46:23.756610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.756593    2559 scope.go:117] "RemoveContainer" containerID="886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d"
Jul 27 16:46:23.756843 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:46:23.756824    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d\": container with ID starting with 886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d not found: ID does not exist" containerID="886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d"
Jul 27 16:46:23.756886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.756850    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d"} err="failed to get container status \"886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d\": rpc error: code = NotFound desc = could not find container \"886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d\": container with ID starting with 886fd4c9472eb0301c3bcdbdf1958b1a454fb799bd1d7b7876e4ecfcf1fc066d not found: ID does not exist"
Jul 27 16:46:23.756886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.756866    2559 scope.go:117] "RemoveContainer" containerID="d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56"
Jul 27 16:46:23.762764 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.762742    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"]
Jul 27 16:46:23.764706 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.764685    2559 scope.go:117] "RemoveContainer" containerID="d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56"
Jul 27 16:46:23.765061 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:46:23.765034    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56\": container with ID starting with d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56 not found: ID does not exist" containerID="d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56"
Jul 27 16:46:23.765175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.765068    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56"} err="failed to get container status \"d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56\": rpc error: code = NotFound desc = could not find container \"d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56\": container with ID starting with d085788f8f47d7f57d441045e7c0f82eecbbf9375071ee9f734ba6f0ce904d56 not found: ID does not exist"
Jul 27 16:46:23.766802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.766784    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a178d363-56d5bf94fc-klh7n"]
Jul 27 16:46:23.779147 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.779123    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"]
Jul 27 16:46:23.783316 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.783294    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a178d363-7dfb95fb7d-jl4c7"]
Jul 27 16:46:23.797406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.797353    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"]
Jul 27 16:46:23.800989 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:23.800967    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a178d363-644ffdbf7b-rchzj"]
Jul 27 16:46:24.735372 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:24.735345    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23bb59fb-8420-4576-b013-c408dbc5b04e" path="/var/lib/kubelet/pods/23bb59fb-8420-4576-b013-c408dbc5b04e/volumes"
Jul 27 16:46:24.735722 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:24.735683    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa621ff-a57b-4263-86d8-9afa702b96d4" path="/var/lib/kubelet/pods/7aa621ff-a57b-4263-86d8-9afa702b96d4/volumes"
Jul 27 16:46:24.735997 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:24.735985    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a184ca07-cfa6-425b-a30b-36531843177f" path="/var/lib/kubelet/pods/a184ca07-cfa6-425b-a30b-36531843177f/volumes"
Jul 27 16:46:27.844218 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:27.844188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:28.020659 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:28.020632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:28.244924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:28.244886    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:28.538234 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:28.538159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:29.066338 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066305    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"]
Jul 27 16:46:29.066669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066604    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bb59fb-8420-4576-b013-c408dbc5b04e" containerName="payload-processing"
Jul 27 16:46:29.066669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066615    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bb59fb-8420-4576-b013-c408dbc5b04e" containerName="payload-processing"
Jul 27 16:46:29.066669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066623    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa621ff-a57b-4263-86d8-9afa702b96d4" containerName="payload-pre-processing"
Jul 27 16:46:29.066669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066630    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa621ff-a57b-4263-86d8-9afa702b96d4" containerName="payload-pre-processing"
Jul 27 16:46:29.066669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066638    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a184ca07-cfa6-425b-a30b-36531843177f" containerName="maas-api"
Jul 27 16:46:29.066669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066643    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a184ca07-cfa6-425b-a30b-36531843177f" containerName="maas-api"
Jul 27 16:46:29.066886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066690    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa621ff-a57b-4263-86d8-9afa702b96d4" containerName="payload-pre-processing"
Jul 27 16:46:29.066886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066697    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a184ca07-cfa6-425b-a30b-36531843177f" containerName="maas-api"
Jul 27 16:46:29.066886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.066703    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bb59fb-8420-4576-b013-c408dbc5b04e" containerName="payload-processing"
Jul 27 16:46:29.069853 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.069832    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.072707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.072689    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a842e2ee-gw-openshift-default-dockercfg-dh544\""
Jul 27 16:46:29.082468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.082448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"]
Jul 27 16:46:29.189217 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sskm6\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-kube-api-access-sskm6\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189374 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-credential-socket\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189374 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-envoy\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189374 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-istio-token\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213339ba-da3b-4d43-89a7-ade77acc3863-istiod-ca-cert\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-socket\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189541 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213339ba-da3b-4d43-89a7-ade77acc3863-istio-podinfo\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-certs\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.189669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.189611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-data\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.263906 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.263869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:29.291032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.290998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-istio-token\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291171 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213339ba-da3b-4d43-89a7-ade77acc3863-istiod-ca-cert\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291171 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-socket\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291171 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213339ba-da3b-4d43-89a7-ade77acc3863-istio-podinfo\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291274 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-certs\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291274 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-data\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sskm6\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-kube-api-access-sskm6\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291349    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-credential-socket\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291408 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-envoy\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-socket\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291614 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-certs\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-credential-socket\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291722 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291653    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-data\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.291812 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.291793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213339ba-da3b-4d43-89a7-ade77acc3863-istiod-ca-cert\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.293546 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.293515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-envoy\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.293661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.293647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213339ba-da3b-4d43-89a7-ade77acc3863-istio-podinfo\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.301254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.301232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-istio-token\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.301365 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.301342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sskm6\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-kube-api-access-sskm6\") pod \"e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") " pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.366584 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.366505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:29.379648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.379603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:29.506001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.505967    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"]
Jul 27 16:46:29.508632 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:46:29.508596    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213339ba_da3b_4d43_89a7_ade77acc3863.slice/crio-a70c7b07806a94c4951d34bf47d5ae05ece4b7a5e51d6759cefdbd1dc129ea8b WatchSource:0}: Error finding container a70c7b07806a94c4951d34bf47d5ae05ece4b7a5e51d6759cefdbd1dc129ea8b: Status 404 returned error can't find the container with id a70c7b07806a94c4951d34bf47d5ae05ece4b7a5e51d6759cefdbd1dc129ea8b
Jul 27 16:46:29.767316 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:29.767262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" event={"ID":"213339ba-da3b-4d43-89a7-ade77acc3863","Type":"ContainerStarted","Data":"a70c7b07806a94c4951d34bf47d5ae05ece4b7a5e51d6759cefdbd1dc129ea8b"}
Jul 27 16:46:32.404872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:32.404840    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:46:32.405122 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:32.404908    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:46:32.405122 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:32.404937    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:46:32.783663 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:32.783566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" event={"ID":"213339ba-da3b-4d43-89a7-ade77acc3863","Type":"ContainerStarted","Data":"5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"}
Jul 27 16:46:32.806779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:32.806727    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" podStartSLOduration=0.912629361 podStartE2EDuration="3.806712671s" totalImagesPullingTime="2.89408331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:46:32.805360851 +0000 UTC m=+1792.706501300" watchObservedRunningTime="2026-07-27 16:46:32.806712671 +0000 UTC m=+1792.707853116"
Jul 27 16:46:33.379975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:33.379936    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:33.385031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:33.385006    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:33.787562 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:33.787530    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:33.788471 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:33.788451    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:34.357094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.357059    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:34.442750 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.442714    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"]
Jul 27 16:46:34.446027 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.446006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.449463 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.449439    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:46:34.449633 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.449439    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:46:34.450567 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.450551    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a842e2ee-serving-cert\""
Jul 27 16:46:34.450635 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.450565    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:46:34.458482 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.458388    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"]
Jul 27 16:46:34.479648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.479614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:34.537993 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.537959    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94627b3c-c3eb-4cb5-9e05-bcf184828373-maas-api-tls\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.537993 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.538002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnzc\" (UniqueName: \"kubernetes.io/projected/94627b3c-c3eb-4cb5-9e05-bcf184828373-kube-api-access-csnzc\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.538223 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.538024    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94627b3c-c3eb-4cb5-9e05-bcf184828373-openshift-service-ca\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.638737 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.638651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94627b3c-c3eb-4cb5-9e05-bcf184828373-maas-api-tls\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.638737 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.638691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnzc\" (UniqueName: \"kubernetes.io/projected/94627b3c-c3eb-4cb5-9e05-bcf184828373-kube-api-access-csnzc\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.638737 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.638713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94627b3c-c3eb-4cb5-9e05-bcf184828373-openshift-service-ca\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.639406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.639385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94627b3c-c3eb-4cb5-9e05-bcf184828373-openshift-service-ca\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.641092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.641068    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94627b3c-c3eb-4cb5-9e05-bcf184828373-maas-api-tls\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.647651 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.647631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnzc\" (UniqueName: \"kubernetes.io/projected/94627b3c-c3eb-4cb5-9e05-bcf184828373-kube-api-access-csnzc\") pod \"maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.656803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.656759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:34.755536 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.755509    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:34.902052 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:34.902026    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"]
Jul 27 16:46:34.904250 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:46:34.904225    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94627b3c_c3eb_4cb5_9e05_bcf184828373.slice/crio-a5391a9a71d5968a216b9d9715fc513b48d0109ccf315ee3b8e4a02508bdddbc WatchSource:0}: Error finding container a5391a9a71d5968a216b9d9715fc513b48d0109ccf315ee3b8e4a02508bdddbc: Status 404 returned error can't find the container with id a5391a9a71d5968a216b9d9715fc513b48d0109ccf315ee3b8e4a02508bdddbc
Jul 27 16:46:35.242844 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.242813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:35.348194 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.348162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:46:35.386990 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.386946    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"]
Jul 27 16:46:35.390231 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.390214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.392920 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.392899    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a842e2ee-dockercfg-sfvtw\""
Jul 27 16:46:35.393145 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.393132    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a842e2ee\""
Jul 27 16:46:35.407780 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.407751    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"]
Jul 27 16:46:35.446334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.446303    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feab17c9-0081-4b9d-b393-e2104a38caba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") " pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.446493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.446358    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7ch\" (UniqueName: \"kubernetes.io/projected/feab17c9-0081-4b9d-b393-e2104a38caba-kube-api-access-sz7ch\") pod \"payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") " pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.547430 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.547351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feab17c9-0081-4b9d-b393-e2104a38caba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") " pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.547430 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.547417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7ch\" (UniqueName: \"kubernetes.io/projected/feab17c9-0081-4b9d-b393-e2104a38caba-kube-api-access-sz7ch\") pod \"payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") " pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.548092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.548072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feab17c9-0081-4b9d-b393-e2104a38caba-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") " pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.556077 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.556055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7ch\" (UniqueName: \"kubernetes.io/projected/feab17c9-0081-4b9d-b393-e2104a38caba-kube-api-access-sz7ch\") pod \"payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") " pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.700711 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.700674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:35.799130 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.799037    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd" event={"ID":"94627b3c-c3eb-4cb5-9e05-bcf184828373","Type":"ContainerStarted","Data":"3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab"}
Jul 27 16:46:35.799130 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.799104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd" event={"ID":"94627b3c-c3eb-4cb5-9e05-bcf184828373","Type":"ContainerStarted","Data":"a5391a9a71d5968a216b9d9715fc513b48d0109ccf315ee3b8e4a02508bdddbc"}
Jul 27 16:46:35.830188 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.830137    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd" podStartSLOduration=1.505378532 podStartE2EDuration="1.830121439s" totalImagesPullingTime="324.742907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:46:35.828472592 +0000 UTC m=+1795.729613039" watchObservedRunningTime="2026-07-27 16:46:35.830121439 +0000 UTC m=+1795.731261947"
Jul 27 16:46:35.843729 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:35.843695    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"]
Jul 27 16:46:35.846505 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:46:35.846482    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeab17c9_0081_4b9d_b393_e2104a38caba.slice/crio-6cf4e6a4768202d159e5b586a460a5b9e35820ee855cb8e6bf130b0a87de2293 WatchSource:0}: Error finding container 6cf4e6a4768202d159e5b586a460a5b9e35820ee855cb8e6bf130b0a87de2293: Status 404 returned error can't find the container with id 6cf4e6a4768202d159e5b586a460a5b9e35820ee855cb8e6bf130b0a87de2293
Jul 27 16:46:36.804272 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:36.804230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj" event={"ID":"feab17c9-0081-4b9d-b393-e2104a38caba","Type":"ContainerStarted","Data":"2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2"}
Jul 27 16:46:36.804712 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:36.804309    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj" event={"ID":"feab17c9-0081-4b9d-b393-e2104a38caba","Type":"ContainerStarted","Data":"6cf4e6a4768202d159e5b586a460a5b9e35820ee855cb8e6bf130b0a87de2293"}
Jul 27 16:46:36.804712 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:36.804386    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:36.822641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:36.822596    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj" podStartSLOduration=1.822581285 podStartE2EDuration="1.822581285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:46:36.821863506 +0000 UTC m=+1796.723003950" watchObservedRunningTime="2026-07-27 16:46:36.822581285 +0000 UTC m=+1796.723721731"
Jul 27 16:46:37.809478 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:37.809448    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:42.814318 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:42.814266    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:46:46.850669 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:46.850633    2559 generic.go:356] "Generic (PLEG): container finished" podID="213339ba-da3b-4d43-89a7-ade77acc3863" containerID="5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2" exitCode=0
Jul 27 16:46:46.851114 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:46.850708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" event={"ID":"213339ba-da3b-4d43-89a7-ade77acc3863","Type":"ContainerDied","Data":"5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"}
Jul 27 16:46:46.851155 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:46.851137    2559 scope.go:117] "RemoveContainer" containerID="5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"
Jul 27 16:46:46.852127 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:46.852102    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:46:46.852181 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:46.852172    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:46:46.852217 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:46.852205    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:46:47.861224 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:47.861188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" event={"ID":"213339ba-da3b-4d43-89a7-ade77acc3863","Type":"ContainerStarted","Data":"82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326"}
Jul 27 16:46:48.380143 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:48.380103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:48.381173 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:48.381149    2559 patch_prober.go:28] interesting pod/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:46:48.381271 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:48.381195    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 27 16:46:48.787806 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:48.787776    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:48.816176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:48.816153    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:46:49.380262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:49.380217    2559 patch_prober.go:28] interesting pod/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:46:49.380784 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:49.380322    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 27 16:46:50.380918 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:50.380882    2559 patch_prober.go:28] interesting pod/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:46:50.381324 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:50.380964    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 27 16:46:51.380709 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:51.380669    2559 patch_prober.go:28] interesting pod/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:46:51.380890 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:51.380738    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 27 16:46:52.383495 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:52.383465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:52.384434 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:52.384413    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:46:54.735703 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:46:54.735667    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:10.096674 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.096627    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"]
Jul 27 16:47:10.097159 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.096891    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd" podUID="94627b3c-c3eb-4cb5-9e05-bcf184828373" containerName="maas-api" containerID="cri-o://3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab" gracePeriod=30
Jul 27 16:47:10.159790 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.159761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:10.188815 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.188785    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"]
Jul 27 16:47:10.189046 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.189021    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj" podUID="feab17c9-0081-4b9d-b393-e2104a38caba" containerName="payload-processing" containerID="cri-o://2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2" gracePeriod=30
Jul 27 16:47:10.251478 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.251450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:10.352137 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.352076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:10.430553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.430530    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:47:10.455148 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.455128    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:47:10.550819 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.550786    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csnzc\" (UniqueName: \"kubernetes.io/projected/94627b3c-c3eb-4cb5-9e05-bcf184828373-kube-api-access-csnzc\") pod \"94627b3c-c3eb-4cb5-9e05-bcf184828373\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") "
Jul 27 16:47:10.550981 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.550877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94627b3c-c3eb-4cb5-9e05-bcf184828373-openshift-service-ca\") pod \"94627b3c-c3eb-4cb5-9e05-bcf184828373\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") "
Jul 27 16:47:10.550981 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.550900    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94627b3c-c3eb-4cb5-9e05-bcf184828373-maas-api-tls\") pod \"94627b3c-c3eb-4cb5-9e05-bcf184828373\" (UID: \"94627b3c-c3eb-4cb5-9e05-bcf184828373\") "
Jul 27 16:47:10.551213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.551188    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94627b3c-c3eb-4cb5-9e05-bcf184828373-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94627b3c-c3eb-4cb5-9e05-bcf184828373" (UID: "94627b3c-c3eb-4cb5-9e05-bcf184828373"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:47:10.552826 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.552805    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94627b3c-c3eb-4cb5-9e05-bcf184828373-kube-api-access-csnzc" (OuterVolumeSpecName: "kube-api-access-csnzc") pod "94627b3c-c3eb-4cb5-9e05-bcf184828373" (UID: "94627b3c-c3eb-4cb5-9e05-bcf184828373"). InnerVolumeSpecName "kube-api-access-csnzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:47:10.552903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.552842    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94627b3c-c3eb-4cb5-9e05-bcf184828373-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94627b3c-c3eb-4cb5-9e05-bcf184828373" (UID: "94627b3c-c3eb-4cb5-9e05-bcf184828373"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:47:10.651344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.651322    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feab17c9-0081-4b9d-b393-e2104a38caba-plugins-config-volume\") pod \"feab17c9-0081-4b9d-b393-e2104a38caba\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") "
Jul 27 16:47:10.651451 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.651410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz7ch\" (UniqueName: \"kubernetes.io/projected/feab17c9-0081-4b9d-b393-e2104a38caba-kube-api-access-sz7ch\") pod \"feab17c9-0081-4b9d-b393-e2104a38caba\" (UID: \"feab17c9-0081-4b9d-b393-e2104a38caba\") "
Jul 27 16:47:10.651553 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.651539    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94627b3c-c3eb-4cb5-9e05-bcf184828373-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:10.651588 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.651557    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94627b3c-c3eb-4cb5-9e05-bcf184828373-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:10.651588 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.651567    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csnzc\" (UniqueName: \"kubernetes.io/projected/94627b3c-c3eb-4cb5-9e05-bcf184828373-kube-api-access-csnzc\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:10.651650 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.651628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feab17c9-0081-4b9d-b393-e2104a38caba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "feab17c9-0081-4b9d-b393-e2104a38caba" (UID: "feab17c9-0081-4b9d-b393-e2104a38caba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:47:10.653171 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.653152    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feab17c9-0081-4b9d-b393-e2104a38caba-kube-api-access-sz7ch" (OuterVolumeSpecName: "kube-api-access-sz7ch") pod "feab17c9-0081-4b9d-b393-e2104a38caba" (UID: "feab17c9-0081-4b9d-b393-e2104a38caba"). InnerVolumeSpecName "kube-api-access-sz7ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:47:10.752871 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.752849    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz7ch\" (UniqueName: \"kubernetes.io/projected/feab17c9-0081-4b9d-b393-e2104a38caba-kube-api-access-sz7ch\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:10.752871 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.752872    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feab17c9-0081-4b9d-b393-e2104a38caba-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:10.963653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.963570    2559 generic.go:356] "Generic (PLEG): container finished" podID="94627b3c-c3eb-4cb5-9e05-bcf184828373" containerID="3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab" exitCode=0
Jul 27 16:47:10.963653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.963628    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"
Jul 27 16:47:10.963846 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.963648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd" event={"ID":"94627b3c-c3eb-4cb5-9e05-bcf184828373","Type":"ContainerDied","Data":"3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab"}
Jul 27 16:47:10.963846 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.963717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd" event={"ID":"94627b3c-c3eb-4cb5-9e05-bcf184828373","Type":"ContainerDied","Data":"a5391a9a71d5968a216b9d9715fc513b48d0109ccf315ee3b8e4a02508bdddbc"}
Jul 27 16:47:10.963846 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.963751    2559 scope.go:117] "RemoveContainer" containerID="3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab"
Jul 27 16:47:10.964985 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.964926    2559 generic.go:356] "Generic (PLEG): container finished" podID="feab17c9-0081-4b9d-b393-e2104a38caba" containerID="2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2" exitCode=0
Jul 27 16:47:10.964985 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.964979    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"
Jul 27 16:47:10.965082 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.964993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj" event={"ID":"feab17c9-0081-4b9d-b393-e2104a38caba","Type":"ContainerDied","Data":"2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2"}
Jul 27 16:47:10.965082 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.965032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj" event={"ID":"feab17c9-0081-4b9d-b393-e2104a38caba","Type":"ContainerDied","Data":"6cf4e6a4768202d159e5b586a460a5b9e35820ee855cb8e6bf130b0a87de2293"}
Jul 27 16:47:10.971626 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.971611    2559 scope.go:117] "RemoveContainer" containerID="3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab"
Jul 27 16:47:10.971845 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:47:10.971830    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab\": container with ID starting with 3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab not found: ID does not exist" containerID="3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab"
Jul 27 16:47:10.971888 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.971853    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab"} err="failed to get container status \"3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab\": rpc error: code = NotFound desc = could not find container \"3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab\": container with ID starting with 3118f7ce2e2983192b26cd47013bc556646100c4807ad97b50e0aec32ed2a8ab not found: ID does not exist"
Jul 27 16:47:10.971888 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.971869    2559 scope.go:117] "RemoveContainer" containerID="2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2"
Jul 27 16:47:10.979056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.979036    2559 scope.go:117] "RemoveContainer" containerID="2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2"
Jul 27 16:47:10.979316 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:47:10.979294    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2\": container with ID starting with 2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2 not found: ID does not exist" containerID="2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2"
Jul 27 16:47:10.979371 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.979327    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2"} err="failed to get container status \"2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2\": rpc error: code = NotFound desc = could not find container \"2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2\": container with ID starting with 2e0c73cc80f04d757e2576a6c5d850789caf4bb517f55944e4bc2dedc6fde3d2 not found: ID does not exist"
Jul 27 16:47:10.982461 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.982440    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"]
Jul 27 16:47:10.987271 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.987244    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a842e2ee-5648f96c-9mpjd"]
Jul 27 16:47:10.999757 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:10.999735    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"]
Jul 27 16:47:11.002304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:11.002266    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a842e2ee-58cb466b8f-979hj"]
Jul 27 16:47:12.735493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:12.735466    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94627b3c-c3eb-4cb5-9e05-bcf184828373" path="/var/lib/kubelet/pods/94627b3c-c3eb-4cb5-9e05-bcf184828373/volumes"
Jul 27 16:47:12.735843 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:12.735777    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feab17c9-0081-4b9d-b393-e2104a38caba" path="/var/lib/kubelet/pods/feab17c9-0081-4b9d-b393-e2104a38caba/volumes"
Jul 27 16:47:14.431195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:14.431160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:14.548447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:14.548414    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"]
Jul 27 16:47:14.548642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:14.548617    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy" containerID="cri-o://82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326" gracePeriod=30
Jul 27 16:47:14.566358 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:14.566333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:19.796737 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.796711    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:47:19.826990 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.826925    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-istio-token\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.826990 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.826964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-data\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827199 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.826999    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sskm6\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-kube-api-access-sskm6\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827199 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827029    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-socket\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827199 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827056    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-certs\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827199 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827106    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213339ba-da3b-4d43-89a7-ade77acc3863-istio-podinfo\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827199 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827173    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-envoy\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827204    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-credential-socket\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827237    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213339ba-da3b-4d43-89a7-ade77acc3863-istiod-ca-cert\") pod \"213339ba-da3b-4d43-89a7-ade77acc3863\" (UID: \"213339ba-da3b-4d43-89a7-ade77acc3863\") "
Jul 27 16:47:19.827477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827315    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:47:19.827649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827555    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.827649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213339ba-da3b-4d43-89a7-ade77acc3863-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:47:19.827753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827671    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-data" (OuterVolumeSpecName: "istio-data") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:47:19.827836 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827814    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:47:19.827902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.827881    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:47:19.829186 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.829165    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-istio-token" (OuterVolumeSpecName: "istio-token") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:47:19.829186 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.829176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-kube-api-access-sskm6" (OuterVolumeSpecName: "kube-api-access-sskm6") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "kube-api-access-sskm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:47:19.829608 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.829589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:47:19.829917 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.829891    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/213339ba-da3b-4d43-89a7-ade77acc3863-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "213339ba-da3b-4d43-89a7-ade77acc3863" (UID: "213339ba-da3b-4d43-89a7-ade77acc3863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:47:19.928619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928591    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213339ba-da3b-4d43-89a7-ade77acc3863-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928616    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928624    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928632    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sskm6\" (UniqueName: \"kubernetes.io/projected/213339ba-da3b-4d43-89a7-ade77acc3863-kube-api-access-sskm6\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928641    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928649    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213339ba-da3b-4d43-89a7-ade77acc3863-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928657    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:19.928818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:19.928664    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213339ba-da3b-4d43-89a7-ade77acc3863-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:47:20.004625 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.004593    2559 generic.go:356] "Generic (PLEG): container finished" podID="213339ba-da3b-4d43-89a7-ade77acc3863" containerID="82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326" exitCode=0
Jul 27 16:47:20.004800 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.004634    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"
Jul 27 16:47:20.004800 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.004639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" event={"ID":"213339ba-da3b-4d43-89a7-ade77acc3863","Type":"ContainerDied","Data":"82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326"}
Jul 27 16:47:20.004800 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.004687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq" event={"ID":"213339ba-da3b-4d43-89a7-ade77acc3863","Type":"ContainerDied","Data":"a70c7b07806a94c4951d34bf47d5ae05ece4b7a5e51d6759cefdbd1dc129ea8b"}
Jul 27 16:47:20.004800 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.004710    2559 scope.go:117] "RemoveContainer" containerID="82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326"
Jul 27 16:47:20.013787 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.013771    2559 scope.go:117] "RemoveContainer" containerID="5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"
Jul 27 16:47:20.021238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.021218    2559 scope.go:117] "RemoveContainer" containerID="82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326"
Jul 27 16:47:20.021494 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:47:20.021477    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326\": container with ID starting with 82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326 not found: ID does not exist" containerID="82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326"
Jul 27 16:47:20.021540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.021502    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326"} err="failed to get container status \"82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326\": rpc error: code = NotFound desc = could not find container \"82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326\": container with ID starting with 82c4865a86e92d191cd0fb68b1d8c25759525e049fb3c1bcf98269dfa291c326 not found: ID does not exist"
Jul 27 16:47:20.021540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.021520    2559 scope.go:117] "RemoveContainer" containerID="5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"
Jul 27 16:47:20.021728 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:47:20.021712    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2\": container with ID starting with 5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2 not found: ID does not exist" containerID="5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"
Jul 27 16:47:20.021769 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.021732    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2"} err="failed to get container status \"5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2\": rpc error: code = NotFound desc = could not find container \"5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2\": container with ID starting with 5e32c04e546440106993902af1ca6f8e5faf28c7f2e83d909a470e200f5179a2 not found: ID does not exist"
Jul 27 16:47:20.029585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.029562    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"]
Jul 27 16:47:20.034430 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.034410    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a842e2ee-gw-openshift-default-68794794c6-z5qzq"]
Jul 27 16:47:20.735370 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:20.735340    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" path="/var/lib/kubelet/pods/213339ba-da3b-4d43-89a7-ade77acc3863/volumes"
Jul 27 16:47:24.837147 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:24.837092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:25.041456 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.041421    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:25.355018 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.354988    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"]
Jul 27 16:47:25.355346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355332    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feab17c9-0081-4b9d-b393-e2104a38caba" containerName="payload-processing"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355347    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="feab17c9-0081-4b9d-b393-e2104a38caba" containerName="payload-processing"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355361    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94627b3c-c3eb-4cb5-9e05-bcf184828373" containerName="maas-api"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355367    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="94627b3c-c3eb-4cb5-9e05-bcf184828373" containerName="maas-api"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355381    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355386    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355393    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy"
Jul 27 16:47:25.355424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355397    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy"
Jul 27 16:47:25.355645 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355444    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="feab17c9-0081-4b9d-b393-e2104a38caba" containerName="payload-processing"
Jul 27 16:47:25.355645 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355452    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="94627b3c-c3eb-4cb5-9e05-bcf184828373" containerName="maas-api"
Jul 27 16:47:25.355645 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.355460    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy"
Jul 27 16:47:25.360010 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.359990    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.363095 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.363076    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5003e71d-openshift-default-dockercfg-nwb54\""
Jul 27 16:47:25.373363 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.373335    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"]
Jul 27 16:47:25.376313 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.376262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:25.472822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.472791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-podinfo\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.472984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.472837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-envoy\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.472984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.472863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-certs\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.472984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.472917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-data\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.472984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.472944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244cc8d-d03c-449a-aa63-dde96dbfc383-istiod-ca-cert\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.473177 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.472991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-token\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.473177 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.473015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-socket\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.473177 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.473035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-credential-socket\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.473177 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.473064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v456f\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-kube-api-access-v456f\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-token\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573626 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-socket\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573626 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-credential-socket\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573626 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v456f\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-kube-api-access-v456f\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573795 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-podinfo\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573841 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-envoy\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573884 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-socket\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573884 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-certs\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-credential-socket\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-data\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.573959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.573930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244cc8d-d03c-449a-aa63-dde96dbfc383-istiod-ca-cert\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.574187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.574154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-data\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.574343 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.574191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-certs\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.574540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.574520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244cc8d-d03c-449a-aa63-dde96dbfc383-istiod-ca-cert\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.575783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.575760    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-envoy\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.576069 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.576040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-podinfo\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.581887 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.581866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-token\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.582090 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.582074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v456f\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-kube-api-access-v456f\") pod \"e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") " pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.669945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.669924    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:25.803551 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.803521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"]
Jul 27 16:47:25.805931 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:47:25.805904    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf244cc8d_d03c_449a_aa63_dde96dbfc383.slice/crio-8cffb8e0df9536fec0f88bd5e36684dab45979d6b541ebd606a6fb03346a96be WatchSource:0}: Error finding container 8cffb8e0df9536fec0f88bd5e36684dab45979d6b541ebd606a6fb03346a96be: Status 404 returned error can't find the container with id 8cffb8e0df9536fec0f88bd5e36684dab45979d6b541ebd606a6fb03346a96be
Jul 27 16:47:25.807983 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.807957    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:47:25.808051 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.808016    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:47:25.808051 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.808044    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:47:25.855850 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:25.855824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:26.032710 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:26.032622    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" event={"ID":"f244cc8d-d03c-449a-aa63-dde96dbfc383","Type":"ContainerStarted","Data":"ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"}
Jul 27 16:47:26.032710 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:26.032684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" event={"ID":"f244cc8d-d03c-449a-aa63-dde96dbfc383","Type":"ContainerStarted","Data":"8cffb8e0df9536fec0f88bd5e36684dab45979d6b541ebd606a6fb03346a96be"}
Jul 27 16:47:26.056529 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:26.056471    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podStartSLOduration=1.056451979 podStartE2EDuration="1.056451979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:47:26.05564432 +0000 UTC m=+1845.956784768" watchObservedRunningTime="2026-07-27 16:47:26.056451979 +0000 UTC m=+1845.957592430"
Jul 27 16:47:26.671196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:26.671158    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:26.676467 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:26.676441    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:27.036390 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.036312    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:27.037351 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.037327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:27.410820 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.410788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:27.552873 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.552835    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:27.639672 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.639640    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"]
Jul 27 16:47:27.640412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.640386    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="213339ba-da3b-4d43-89a7-ade77acc3863" containerName="istio-proxy"
Jul 27 16:47:27.645033 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.645009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.648064 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.648039    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5003e71d-serving-cert\""
Jul 27 16:47:27.648165 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.648070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:47:27.648165 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.648073    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:47:27.649238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.649219    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:47:27.657259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.657238    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"]
Jul 27 16:47:27.681334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.681240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:27.693555 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.693528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6800030e-5368-4dbd-a88f-10355d1ee987-openshift-service-ca\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.693692 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.693592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdzd\" (UniqueName: \"kubernetes.io/projected/6800030e-5368-4dbd-a88f-10355d1ee987-kube-api-access-dxdzd\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.693692 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.693636    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6800030e-5368-4dbd-a88f-10355d1ee987-maas-api-tls\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.776173 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.776141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:27.794298 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.794248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6800030e-5368-4dbd-a88f-10355d1ee987-openshift-service-ca\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.794435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.794316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdzd\" (UniqueName: \"kubernetes.io/projected/6800030e-5368-4dbd-a88f-10355d1ee987-kube-api-access-dxdzd\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.794435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.794357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6800030e-5368-4dbd-a88f-10355d1ee987-maas-api-tls\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.794945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.794921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6800030e-5368-4dbd-a88f-10355d1ee987-openshift-service-ca\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.796666 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.796647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6800030e-5368-4dbd-a88f-10355d1ee987-maas-api-tls\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.805628 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.805604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdzd\" (UniqueName: \"kubernetes.io/projected/6800030e-5368-4dbd-a88f-10355d1ee987-kube-api-access-dxdzd\") pod \"maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:27.956534 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:27.956463    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:28.084568 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.084536    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"]
Jul 27 16:47:28.086718 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:47:28.086687    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6800030e_5368_4dbd_a88f_10355d1ee987.slice/crio-b8f0a371811601d880f98d7c3b8a52c744d483ec67a5ec52d751a1ac7b46f15c WatchSource:0}: Error finding container b8f0a371811601d880f98d7c3b8a52c744d483ec67a5ec52d751a1ac7b46f15c: Status 404 returned error can't find the container with id b8f0a371811601d880f98d7c3b8a52c744d483ec67a5ec52d751a1ac7b46f15c
Jul 27 16:47:28.234398 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.234317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:28.331160 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.331124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:47:28.350808 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.350777    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"]
Jul 27 16:47:28.355291 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.355261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.358468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.358445    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5003e71d\""
Jul 27 16:47:28.358592 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.358448    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5003e71d-dockercfg-65g2c\""
Jul 27 16:47:28.358925 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.358906    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"]
Jul 27 16:47:28.361804 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.361783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.369031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.369014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"]
Jul 27 16:47:28.380392 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.380373    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"]
Jul 27 16:47:28.400683 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.400655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a862b-a6fe-43b3-9aed-5e45b91602c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") " pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.400803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.400692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n428\" (UniqueName: \"kubernetes.io/projected/741a862b-a6fe-43b3-9aed-5e45b91602c0-kube-api-access-2n428\") pod \"payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") " pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.400803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.400757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026d5495-1f10-4f14-90ef-12e6c50839c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.400803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.400789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9686\" (UniqueName: \"kubernetes.io/projected/026d5495-1f10-4f14-90ef-12e6c50839c8-kube-api-access-w9686\") pod \"payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.502376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.502249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026d5495-1f10-4f14-90ef-12e6c50839c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.502376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.502355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9686\" (UniqueName: \"kubernetes.io/projected/026d5495-1f10-4f14-90ef-12e6c50839c8-kube-api-access-w9686\") pod \"payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.502615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.502454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a862b-a6fe-43b3-9aed-5e45b91602c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") " pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.502615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.502493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n428\" (UniqueName: \"kubernetes.io/projected/741a862b-a6fe-43b3-9aed-5e45b91602c0-kube-api-access-2n428\") pod \"payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") " pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.503172 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.503148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026d5495-1f10-4f14-90ef-12e6c50839c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.503239 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.503228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a862b-a6fe-43b3-9aed-5e45b91602c0-plugins-config-volume\") pod \"payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") " pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.512148 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.512126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n428\" (UniqueName: \"kubernetes.io/projected/741a862b-a6fe-43b3-9aed-5e45b91602c0-kube-api-access-2n428\") pod \"payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") " pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.512268 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.512174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9686\" (UniqueName: \"kubernetes.io/projected/026d5495-1f10-4f14-90ef-12e6c50839c8-kube-api-access-w9686\") pod \"payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.666295 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.666259    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:28.674196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.674173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:28.804205 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.804172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"]
Jul 27 16:47:28.808795 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:47:28.808766    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026d5495_1f10_4f14_90ef_12e6c50839c8.slice/crio-7b26153554cad900b0c42f2f57091335186238076562f6305bdf7866c8a01c32 WatchSource:0}: Error finding container 7b26153554cad900b0c42f2f57091335186238076562f6305bdf7866c8a01c32: Status 404 returned error can't find the container with id 7b26153554cad900b0c42f2f57091335186238076562f6305bdf7866c8a01c32
Jul 27 16:47:28.828176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:28.828152    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"]
Jul 27 16:47:28.831268 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:47:28.831246    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741a862b_a6fe_43b3_9aed_5e45b91602c0.slice/crio-38dd0d0dc15808c70536cf93c063db3f3961106642d485c0ce0e38a9b35cdc81 WatchSource:0}: Error finding container 38dd0d0dc15808c70536cf93c063db3f3961106642d485c0ce0e38a9b35cdc81: Status 404 returned error can't find the container with id 38dd0d0dc15808c70536cf93c063db3f3961106642d485c0ce0e38a9b35cdc81
Jul 27 16:47:29.049124 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.049021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq" event={"ID":"026d5495-1f10-4f14-90ef-12e6c50839c8","Type":"ContainerStarted","Data":"3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc"}
Jul 27 16:47:29.049124 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.049087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq" event={"ID":"026d5495-1f10-4f14-90ef-12e6c50839c8","Type":"ContainerStarted","Data":"7b26153554cad900b0c42f2f57091335186238076562f6305bdf7866c8a01c32"}
Jul 27 16:47:29.049124 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.049122    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:29.050802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.050770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj" event={"ID":"6800030e-5368-4dbd-a88f-10355d1ee987","Type":"ContainerStarted","Data":"c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743"}
Jul 27 16:47:29.050942 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.050826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:29.050942 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.050849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj" event={"ID":"6800030e-5368-4dbd-a88f-10355d1ee987","Type":"ContainerStarted","Data":"b8f0a371811601d880f98d7c3b8a52c744d483ec67a5ec52d751a1ac7b46f15c"}
Jul 27 16:47:29.052301 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.052256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c" event={"ID":"741a862b-a6fe-43b3-9aed-5e45b91602c0","Type":"ContainerStarted","Data":"65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b"}
Jul 27 16:47:29.052420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.052328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c" event={"ID":"741a862b-a6fe-43b3-9aed-5e45b91602c0","Type":"ContainerStarted","Data":"38dd0d0dc15808c70536cf93c063db3f3961106642d485c0ce0e38a9b35cdc81"}
Jul 27 16:47:29.067479 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.067434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq" podStartSLOduration=1.067419527 podStartE2EDuration="1.067419527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:47:29.065939387 +0000 UTC m=+1848.967079832" watchObservedRunningTime="2026-07-27 16:47:29.067419527 +0000 UTC m=+1848.968559968"
Jul 27 16:47:29.084471 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.084434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c" podStartSLOduration=1.084419783 podStartE2EDuration="1.084419783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:47:29.082900619 +0000 UTC m=+1848.984041068" watchObservedRunningTime="2026-07-27 16:47:29.084419783 +0000 UTC m=+1848.985560229"
Jul 27 16:47:29.104082 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:29.104044    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj" podStartSLOduration=1.640412016 podStartE2EDuration="2.10403072s" totalImagesPullingTime="463.618704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:47:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:47:29.10311777 +0000 UTC m=+1849.004258216" watchObservedRunningTime="2026-07-27 16:47:29.10403072 +0000 UTC m=+1849.005171166"
Jul 27 16:47:30.057333 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:30.057303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:35.062875 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:35.062842    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:47:40.059497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:40.059464    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:47:41.064876 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:41.064794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:47:42.037544 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.037462    2559 patch_prober.go:28] interesting pod/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:47:42.037544 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.037532    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 27 16:47:42.119614 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.119581    2559 generic.go:356] "Generic (PLEG): container finished" podID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerID="ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90" exitCode=0
Jul 27 16:47:42.119975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.119649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" event={"ID":"f244cc8d-d03c-449a-aa63-dde96dbfc383","Type":"ContainerDied","Data":"ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"}
Jul 27 16:47:42.120021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.119997    2559 scope.go:117] "RemoveContainer" containerID="ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"
Jul 27 16:47:42.120879 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.120855    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:47:42.120932 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.120919    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:47:42.120968 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:42.120946    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:47:43.124925 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:43.124890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" event={"ID":"f244cc8d-d03c-449a-aa63-dde96dbfc383","Type":"ContainerStarted","Data":"a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e"}
Jul 27 16:47:43.670937 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:43.670907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:43.671123 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:43.671106    2559 patch_prober.go:28] interesting pod/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:47:43.671167 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:43.671144    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 27 16:47:44.671117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:44.671078    2559 patch_prober.go:28] interesting pod/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:47:44.671517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:44.671139    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 27 16:47:45.670984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:45.670939    2559 patch_prober.go:28] interesting pod/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:47:45.671187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:45.671004    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 27 16:47:46.670666 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:46.670624    2559 patch_prober.go:28] interesting pod/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:47:46.670868 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:46.670688    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Jul 27 16:47:47.674032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:47.673997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:47.674385 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:47.674265    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:47:47.674838 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:47:47.674818    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:48:03.788884 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.788809    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"]
Jul 27 16:48:03.789366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.789009    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj" podUID="6800030e-5368-4dbd-a88f-10355d1ee987" containerName="maas-api" containerID="cri-o://c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743" gracePeriod=30
Jul 27 16:48:03.822758 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.822728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:03.846997 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.846959    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"]
Jul 27 16:48:03.847292 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.847251    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c" podUID="741a862b-a6fe-43b3-9aed-5e45b91602c0" containerName="payload-processing" containerID="cri-o://65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b" gracePeriod=30
Jul 27 16:48:03.854049 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.854023    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"]
Jul 27 16:48:03.854229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:03.854209    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq" podUID="026d5495-1f10-4f14-90ef-12e6c50839c8" containerName="payload-pre-processing" containerID="cri-o://3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc" gracePeriod=30
Jul 27 16:48:04.096820 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.096795    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:48:04.111535 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.111505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6800030e-5368-4dbd-a88f-10355d1ee987-maas-api-tls\") pod \"6800030e-5368-4dbd-a88f-10355d1ee987\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") "
Jul 27 16:48:04.111682 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.111569    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6800030e-5368-4dbd-a88f-10355d1ee987-openshift-service-ca\") pod \"6800030e-5368-4dbd-a88f-10355d1ee987\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") "
Jul 27 16:48:04.111682 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.111632    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxdzd\" (UniqueName: \"kubernetes.io/projected/6800030e-5368-4dbd-a88f-10355d1ee987-kube-api-access-dxdzd\") pod \"6800030e-5368-4dbd-a88f-10355d1ee987\" (UID: \"6800030e-5368-4dbd-a88f-10355d1ee987\") "
Jul 27 16:48:04.111949 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.111924    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6800030e-5368-4dbd-a88f-10355d1ee987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6800030e-5368-4dbd-a88f-10355d1ee987" (UID: "6800030e-5368-4dbd-a88f-10355d1ee987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:48:04.114483 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.114455    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6800030e-5368-4dbd-a88f-10355d1ee987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6800030e-5368-4dbd-a88f-10355d1ee987" (UID: "6800030e-5368-4dbd-a88f-10355d1ee987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:48:04.114584 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.114496    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6800030e-5368-4dbd-a88f-10355d1ee987-kube-api-access-dxdzd" (OuterVolumeSpecName: "kube-api-access-dxdzd") pod "6800030e-5368-4dbd-a88f-10355d1ee987" (UID: "6800030e-5368-4dbd-a88f-10355d1ee987"). InnerVolumeSpecName "kube-api-access-dxdzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:48:04.146710 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.146437    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:48:04.164015 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.163987    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:48:04.212078 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212047    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a862b-a6fe-43b3-9aed-5e45b91602c0-plugins-config-volume\") pod \"741a862b-a6fe-43b3-9aed-5e45b91602c0\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") "
Jul 27 16:48:04.212233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212096    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n428\" (UniqueName: \"kubernetes.io/projected/741a862b-a6fe-43b3-9aed-5e45b91602c0-kube-api-access-2n428\") pod \"741a862b-a6fe-43b3-9aed-5e45b91602c0\" (UID: \"741a862b-a6fe-43b3-9aed-5e45b91602c0\") "
Jul 27 16:48:04.212233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212125    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9686\" (UniqueName: \"kubernetes.io/projected/026d5495-1f10-4f14-90ef-12e6c50839c8-kube-api-access-w9686\") pod \"026d5495-1f10-4f14-90ef-12e6c50839c8\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") "
Jul 27 16:48:04.212233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212224    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026d5495-1f10-4f14-90ef-12e6c50839c8-plugins-config-volume\") pod \"026d5495-1f10-4f14-90ef-12e6c50839c8\" (UID: \"026d5495-1f10-4f14-90ef-12e6c50839c8\") "
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212504    2559 generic.go:356] "Generic (PLEG): container finished" podID="6800030e-5368-4dbd-a88f-10355d1ee987" containerID="c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743" exitCode=0
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj" event={"ID":"6800030e-5368-4dbd-a88f-10355d1ee987","Type":"ContainerDied","Data":"c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743"}
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212578    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj" event={"ID":"6800030e-5368-4dbd-a88f-10355d1ee987","Type":"ContainerDied","Data":"b8f0a371811601d880f98d7c3b8a52c744d483ec67a5ec52d751a1ac7b46f15c"}
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212512    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdzd\" (UniqueName: \"kubernetes.io/projected/6800030e-5368-4dbd-a88f-10355d1ee987-kube-api-access-dxdzd\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212647    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6800030e-5368-4dbd-a88f-10355d1ee987-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741a862b-a6fe-43b3-9aed-5e45b91602c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "741a862b-a6fe-43b3-9aed-5e45b91602c0" (UID: "741a862b-a6fe-43b3-9aed-5e45b91602c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212663    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6800030e-5368-4dbd-a88f-10355d1ee987-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212623    2559 scope.go:117] "RemoveContainer" containerID="c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743"
Jul 27 16:48:04.213087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.212616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026d5495-1f10-4f14-90ef-12e6c50839c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "026d5495-1f10-4f14-90ef-12e6c50839c8" (UID: "026d5495-1f10-4f14-90ef-12e6c50839c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:48:04.214460 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.214419    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741a862b-a6fe-43b3-9aed-5e45b91602c0-kube-api-access-2n428" (OuterVolumeSpecName: "kube-api-access-2n428") pod "741a862b-a6fe-43b3-9aed-5e45b91602c0" (UID: "741a862b-a6fe-43b3-9aed-5e45b91602c0"). InnerVolumeSpecName "kube-api-access-2n428". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:48:04.214858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.214816    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026d5495-1f10-4f14-90ef-12e6c50839c8-kube-api-access-w9686" (OuterVolumeSpecName: "kube-api-access-w9686") pod "026d5495-1f10-4f14-90ef-12e6c50839c8" (UID: "026d5495-1f10-4f14-90ef-12e6c50839c8"). InnerVolumeSpecName "kube-api-access-w9686". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:48:04.215187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.215162    2559 generic.go:356] "Generic (PLEG): container finished" podID="741a862b-a6fe-43b3-9aed-5e45b91602c0" containerID="65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b" exitCode=0
Jul 27 16:48:04.215343 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.215201    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"
Jul 27 16:48:04.215343 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.215253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c" event={"ID":"741a862b-a6fe-43b3-9aed-5e45b91602c0","Type":"ContainerDied","Data":"65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b"}
Jul 27 16:48:04.215343 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.215315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c" event={"ID":"741a862b-a6fe-43b3-9aed-5e45b91602c0","Type":"ContainerDied","Data":"38dd0d0dc15808c70536cf93c063db3f3961106642d485c0ce0e38a9b35cdc81"}
Jul 27 16:48:04.217041 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.217018    2559 generic.go:356] "Generic (PLEG): container finished" podID="026d5495-1f10-4f14-90ef-12e6c50839c8" containerID="3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc" exitCode=0
Jul 27 16:48:04.217196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.217046    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"
Jul 27 16:48:04.217196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.217063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq" event={"ID":"026d5495-1f10-4f14-90ef-12e6c50839c8","Type":"ContainerDied","Data":"3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc"}
Jul 27 16:48:04.217196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.217100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq" event={"ID":"026d5495-1f10-4f14-90ef-12e6c50839c8","Type":"ContainerDied","Data":"7b26153554cad900b0c42f2f57091335186238076562f6305bdf7866c8a01c32"}
Jul 27 16:48:04.224659 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.224641    2559 scope.go:117] "RemoveContainer" containerID="c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743"
Jul 27 16:48:04.225091 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:48:04.225059    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743\": container with ID starting with c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743 not found: ID does not exist" containerID="c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743"
Jul 27 16:48:04.225215 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.225097    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743"} err="failed to get container status \"c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743\": rpc error: code = NotFound desc = could not find container \"c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743\": container with ID starting with c627925a0fd00ce6cd5f970834c9ecebb654c29022aa654c86f6e3f071071743 not found: ID does not exist"
Jul 27 16:48:04.225215 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.225114    2559 scope.go:117] "RemoveContainer" containerID="65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b"
Jul 27 16:48:04.233409 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.233392    2559 scope.go:117] "RemoveContainer" containerID="65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b"
Jul 27 16:48:04.233665 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:48:04.233647    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b\": container with ID starting with 65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b not found: ID does not exist" containerID="65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b"
Jul 27 16:48:04.233714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.233672    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b"} err="failed to get container status \"65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b\": rpc error: code = NotFound desc = could not find container \"65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b\": container with ID starting with 65ee95e76021e7639503d27149e54e02159a3050eafd776c9f488417e20cbb5b not found: ID does not exist"
Jul 27 16:48:04.233714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.233688    2559 scope.go:117] "RemoveContainer" containerID="3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc"
Jul 27 16:48:04.240826 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.240807    2559 scope.go:117] "RemoveContainer" containerID="3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc"
Jul 27 16:48:04.241086 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:48:04.241061    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc\": container with ID starting with 3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc not found: ID does not exist" containerID="3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc"
Jul 27 16:48:04.241174 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.241091    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc"} err="failed to get container status \"3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc\": rpc error: code = NotFound desc = could not find container \"3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc\": container with ID starting with 3add7efa0405fe146b4c11a49ea15cdba74dd51cd9143de2a60c4281e30380fc not found: ID does not exist"
Jul 27 16:48:04.241502 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.241484    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"]
Jul 27 16:48:04.246075 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.246054    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5003e71d-7fbb697cc5-f9mwj"]
Jul 27 16:48:04.258023 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.258003    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"]
Jul 27 16:48:04.264766 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.264745    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5003e71d-9c6c95f58-25l9c"]
Jul 27 16:48:04.278149 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.278128    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"]
Jul 27 16:48:04.282416 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.282397    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5003e71d-7b5c8557dd-846mq"]
Jul 27 16:48:04.313477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.313456    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/741a862b-a6fe-43b3-9aed-5e45b91602c0-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.313477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.313477    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n428\" (UniqueName: \"kubernetes.io/projected/741a862b-a6fe-43b3-9aed-5e45b91602c0-kube-api-access-2n428\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.313609 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.313487    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9686\" (UniqueName: \"kubernetes.io/projected/026d5495-1f10-4f14-90ef-12e6c50839c8-kube-api-access-w9686\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.313609 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.313496    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026d5495-1f10-4f14-90ef-12e6c50839c8-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:04.736455 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.736418    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="026d5495-1f10-4f14-90ef-12e6c50839c8" path="/var/lib/kubelet/pods/026d5495-1f10-4f14-90ef-12e6c50839c8/volumes"
Jul 27 16:48:04.736795 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.736782    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6800030e-5368-4dbd-a88f-10355d1ee987" path="/var/lib/kubelet/pods/6800030e-5368-4dbd-a88f-10355d1ee987/volumes"
Jul 27 16:48:04.737075 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.737064    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="741a862b-a6fe-43b3-9aed-5e45b91602c0" path="/var/lib/kubelet/pods/741a862b-a6fe-43b3-9aed-5e45b91602c0/volumes"
Jul 27 16:48:04.794117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.794082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:04.855508 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:04.855472    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:08.600100 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:08.600067    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:09.602101 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:09.602060    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"]
Jul 27 16:48:09.602567 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:09.602336    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy" containerID="cri-o://a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e" gracePeriod=30
Jul 27 16:48:09.612707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:09.612676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:09.857417 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:09.857328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:10.086459 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.086420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:10.384772 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.384741    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"]
Jul 27 16:48:10.385046 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385033    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741a862b-a6fe-43b3-9aed-5e45b91602c0" containerName="payload-processing"
Jul 27 16:48:10.385046 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385046    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="741a862b-a6fe-43b3-9aed-5e45b91602c0" containerName="payload-processing"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385059    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="026d5495-1f10-4f14-90ef-12e6c50839c8" containerName="payload-pre-processing"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385065    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="026d5495-1f10-4f14-90ef-12e6c50839c8" containerName="payload-pre-processing"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385073    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6800030e-5368-4dbd-a88f-10355d1ee987" containerName="maas-api"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385079    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6800030e-5368-4dbd-a88f-10355d1ee987" containerName="maas-api"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385135    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="741a862b-a6fe-43b3-9aed-5e45b91602c0" containerName="payload-processing"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385143    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6800030e-5368-4dbd-a88f-10355d1ee987" containerName="maas-api"
Jul 27 16:48:10.385162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.385150    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="026d5495-1f10-4f14-90ef-12e6c50839c8" containerName="payload-pre-processing"
Jul 27 16:48:10.388504 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.388487    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.391871 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.391854    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-370eeced-openshift-default-dockercfg-2b8q7\""
Jul 27 16:48:10.404716 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.404664    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"]
Jul 27 16:48:10.426693 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.426660    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:10.464362 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-data\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464362 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e448cf-02e1-47b2-bc79-4e226f7756c0-istiod-ca-cert\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-credential-socket\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvpd\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-kube-api-access-ktvpd\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-podinfo\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-socket\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-envoy\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-certs\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.464571 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.464565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-token\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565524 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-podinfo\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565524 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-socket\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565798 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-envoy\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565798 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-certs\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565798 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-token\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565798 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-data\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.565798 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e448cf-02e1-47b2-bc79-4e226f7756c0-istiod-ca-cert\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566059 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-credential-socket\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566059 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvpd\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-kube-api-access-ktvpd\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566059 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.565971    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-socket\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566059 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.566043    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-certs\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566217 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.566091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-data\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566217 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.566202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-credential-socket\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.566560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.566535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e448cf-02e1-47b2-bc79-4e226f7756c0-istiod-ca-cert\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.568010 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.567980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-envoy\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.568151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.568135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-podinfo\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.584343 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.584320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-token\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.584558 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.584536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvpd\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-kube-api-access-ktvpd\") pod \"e2e-derive-370eeced-openshift-default-747764785b-lbdbh\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") " pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.694666 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.694632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:10.698683 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.698664    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:10.843483 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.843449    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"]
Jul 27 16:48:10.846650 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:48:10.846624    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e448cf_02e1_47b2_bc79_4e226f7756c0.slice/crio-186d51c7ded412ff55db732831f0e408059fa91f6b76d08f7cdb1a367c80afa8 WatchSource:0}: Error finding container 186d51c7ded412ff55db732831f0e408059fa91f6b76d08f7cdb1a367c80afa8: Status 404 returned error can't find the container with id 186d51c7ded412ff55db732831f0e408059fa91f6b76d08f7cdb1a367c80afa8
Jul 27 16:48:10.848573 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.848544    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:48:10.848642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.848604    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:48:10.848642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:10.848632    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:48:11.250993 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:11.250954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" event={"ID":"20e448cf-02e1-47b2-bc79-4e226f7756c0","Type":"ContainerStarted","Data":"f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"}
Jul 27 16:48:11.251212 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:11.251032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" event={"ID":"20e448cf-02e1-47b2-bc79-4e226f7756c0","Type":"ContainerStarted","Data":"186d51c7ded412ff55db732831f0e408059fa91f6b76d08f7cdb1a367c80afa8"}
Jul 27 16:48:11.290397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:11.290347    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" podStartSLOduration=1.290332765 podStartE2EDuration="1.290332765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:48:11.288504841 +0000 UTC m=+1891.189645298" watchObservedRunningTime="2026-07-27 16:48:11.290332765 +0000 UTC m=+1891.191473211"
Jul 27 16:48:11.699346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:11.699313    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:11.703956 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:11.703930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:12.255540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:12.255507    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:12.256381 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:12.256365    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:12.888362 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:12.888327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:13.012483 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.012452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:13.224489 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.224458    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"]
Jul 27 16:48:13.227606 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.227589    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.232462 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.232445    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:48:13.232560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.232487    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:48:13.232830 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.232813    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-370eeced-serving-cert\""
Jul 27 16:48:13.232894 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.232836    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:48:13.251203 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.251178    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"]
Jul 27 16:48:13.287802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.287771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc88x\" (UniqueName: \"kubernetes.io/projected/5ba1af96-d868-4489-aa91-7997faa14f06-kube-api-access-sc88x\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.288001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.287985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba1af96-d868-4489-aa91-7997faa14f06-maas-api-tls\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.288098 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.288010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1af96-d868-4489-aa91-7997faa14f06-openshift-service-ca\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.312758 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.312728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:13.389402 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.389373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba1af96-d868-4489-aa91-7997faa14f06-maas-api-tls\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.389402 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.389408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1af96-d868-4489-aa91-7997faa14f06-openshift-service-ca\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.389619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.389536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc88x\" (UniqueName: \"kubernetes.io/projected/5ba1af96-d868-4489-aa91-7997faa14f06-kube-api-access-sc88x\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.390020 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.390000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1af96-d868-4489-aa91-7997faa14f06-openshift-service-ca\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.391723 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.391702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba1af96-d868-4489-aa91-7997faa14f06-maas-api-tls\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.404318 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.404297    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc88x\" (UniqueName: \"kubernetes.io/projected/5ba1af96-d868-4489-aa91-7997faa14f06-kube-api-access-sc88x\") pod \"maas-api-e2e-derive-370eeced-8656bfdc67-44nd2\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.428157 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.428133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:13.537808 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.537725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:13.671864 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:13.671142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"]
Jul 27 16:48:13.674040 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:48:13.674013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba1af96_d868_4489_aa91_7997faa14f06.slice/crio-6f4df0daf80c11220818f6e94d4e2a45a1410afb5b21c317b2f8c97ec6fe4d7e WatchSource:0}: Error finding container 6f4df0daf80c11220818f6e94d4e2a45a1410afb5b21c317b2f8c97ec6fe4d7e: Status 404 returned error can't find the container with id 6f4df0daf80c11220818f6e94d4e2a45a1410afb5b21c317b2f8c97ec6fe4d7e
Jul 27 16:48:14.069869 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.069839    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:14.174477 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.174450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:14.266567 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.266529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2" event={"ID":"5ba1af96-d868-4489-aa91-7997faa14f06","Type":"ContainerStarted","Data":"bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805"}
Jul 27 16:48:14.266742 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.266593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2" event={"ID":"5ba1af96-d868-4489-aa91-7997faa14f06","Type":"ContainerStarted","Data":"6f4df0daf80c11220818f6e94d4e2a45a1410afb5b21c317b2f8c97ec6fe4d7e"}
Jul 27 16:48:14.266913 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.266896    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:14.301902 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.301856    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2" podStartSLOduration=0.867786923 podStartE2EDuration="1.301842369s" totalImagesPullingTime="434.055446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:48:14.298941667 +0000 UTC m=+1894.200082123" watchObservedRunningTime="2026-07-27 16:48:14.301842369 +0000 UTC m=+1894.202982815"
Jul 27 16:48:14.856714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.856691    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:48:14.902447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902421    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-podinfo\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902451    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244cc8d-d03c-449a-aa63-dde96dbfc383-istiod-ca-cert\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902498    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-certs\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902518    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-envoy\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902553    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v456f\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-kube-api-access-v456f\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-data\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902606    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-credential-socket\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-token\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.902679 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-socket\") pod \"f244cc8d-d03c-449a-aa63-dde96dbfc383\" (UID: \"f244cc8d-d03c-449a-aa63-dde96dbfc383\") "
Jul 27 16:48:14.903066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.902826    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:48:14.903066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.903056    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:48:14.903162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.903094    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f244cc8d-d03c-449a-aa63-dde96dbfc383-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:48:14.903296 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.903259    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:48:14.903753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.903727    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-data" (OuterVolumeSpecName: "istio-data") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:48:14.904751 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.904724    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:48:14.904973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.904946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-token" (OuterVolumeSpecName: "istio-token") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:48:14.904973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.904961    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-kube-api-access-v456f" (OuterVolumeSpecName: "kube-api-access-v456f") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "kube-api-access-v456f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:48:14.905083 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:14.905065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f244cc8d-d03c-449a-aa63-dde96dbfc383" (UID: "f244cc8d-d03c-449a-aa63-dde96dbfc383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:48:15.003648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003613    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003641    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003650    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f244cc8d-d03c-449a-aa63-dde96dbfc383-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003659    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003667    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003675    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v456f\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-kube-api-access-v456f\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003683    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003691    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f244cc8d-d03c-449a-aa63-dde96dbfc383-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.003895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.003699    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f244cc8d-d03c-449a-aa63-dde96dbfc383-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:15.272688 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.272589    2559 generic.go:356] "Generic (PLEG): container finished" podID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerID="a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e" exitCode=0
Jul 27 16:48:15.272688 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.272644    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"
Jul 27 16:48:15.272688 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.272679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" event={"ID":"f244cc8d-d03c-449a-aa63-dde96dbfc383","Type":"ContainerDied","Data":"a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e"}
Jul 27 16:48:15.273243 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.272728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl" event={"ID":"f244cc8d-d03c-449a-aa63-dde96dbfc383","Type":"ContainerDied","Data":"8cffb8e0df9536fec0f88bd5e36684dab45979d6b541ebd606a6fb03346a96be"}
Jul 27 16:48:15.273243 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.272755    2559 scope.go:117] "RemoveContainer" containerID="a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e"
Jul 27 16:48:15.282358 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.282337    2559 scope.go:117] "RemoveContainer" containerID="ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"
Jul 27 16:48:15.291032 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.291013    2559 scope.go:117] "RemoveContainer" containerID="a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e"
Jul 27 16:48:15.291310 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:48:15.291265    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e\": container with ID starting with a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e not found: ID does not exist" containerID="a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e"
Jul 27 16:48:15.291405 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.291321    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e"} err="failed to get container status \"a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e\": rpc error: code = NotFound desc = could not find container \"a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e\": container with ID starting with a117d52c3b6fb9cc4e11aa1098cd94bba4ebaba763cc436a29fadea0445e1a8e not found: ID does not exist"
Jul 27 16:48:15.291405 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.291346    2559 scope.go:117] "RemoveContainer" containerID="ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"
Jul 27 16:48:15.291590 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:48:15.291574    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90\": container with ID starting with ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90 not found: ID does not exist" containerID="ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"
Jul 27 16:48:15.291642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.291597    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90"} err="failed to get container status \"ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90\": rpc error: code = NotFound desc = could not find container \"ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90\": container with ID starting with ea39402e4b43c352540d3113be1894f0b0f2a170badfb070f6a92d52bcb35d90 not found: ID does not exist"
Jul 27 16:48:15.302635 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.302611    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"]
Jul 27 16:48:15.308150 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:15.308129    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5003e71d-openshift-default-5bb5b8f8b-r77fl"]
Jul 27 16:48:16.736101 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:16.736065    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" path="/var/lib/kubelet/pods/f244cc8d-d03c-449a-aa63-dde96dbfc383/volumes"
Jul 27 16:48:21.283814 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:21.283786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:24.312461 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:24.312379    2559 generic.go:356] "Generic (PLEG): container finished" podID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerID="f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd" exitCode=0
Jul 27 16:48:24.312840 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:24.312452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" event={"ID":"20e448cf-02e1-47b2-bc79-4e226f7756c0","Type":"ContainerDied","Data":"f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"}
Jul 27 16:48:24.312890 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:24.312865    2559 scope.go:117] "RemoveContainer" containerID="f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"
Jul 27 16:48:24.313863 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:24.313835    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:48:24.313927 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:24.313900    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:48:24.313967 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:24.313936    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:48:25.318572 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:25.318541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" event={"ID":"20e448cf-02e1-47b2-bc79-4e226f7756c0","Type":"ContainerStarted","Data":"f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8"}
Jul 27 16:48:25.698893 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:25.698851    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:25.700055 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:25.700025    2559 patch_prober.go:28] interesting pod/e2e-derive-370eeced-openshift-default-747764785b-lbdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 27 16:48:25.700182 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:25.700078    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 27 16:48:26.699619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:26.699574    2559 patch_prober.go:28] interesting pod/e2e-derive-370eeced-openshift-default-747764785b-lbdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 27 16:48:26.700092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:26.699650    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 27 16:48:27.255971 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:27.255926    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:27.699050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:27.699016    2559 patch_prober.go:28] interesting pod/e2e-derive-370eeced-openshift-default-747764785b-lbdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 27 16:48:27.699243 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:27.699071    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 27 16:48:28.699767 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:28.699723    2559 patch_prober.go:28] interesting pod/e2e-derive-370eeced-openshift-default-747764785b-lbdbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 27 16:48:28.700161 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:28.699800    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 27 16:48:29.498089 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:29.498058    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:29.602047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:29.602015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:29.702627 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:29.702597    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:29.703545 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:29.703524    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:48:29.920433 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:29.920397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:30.475729 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:30.475699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:30.755149 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:30.755060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:30.860845 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:30.860812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:31.716691 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:31.716657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:34.295081 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:34.295048    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:35.925863 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:35.925828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:36.483354 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:36.483323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:37.061024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:37.060995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:37.678645 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:37.678614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:40.410797 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:40.410757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:42.279221 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:42.279185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:43.685455 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:43.685407    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:47.961069 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:47.961036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:50.057962 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.057932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:50.605694 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.605654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:50.837834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.837800    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"]
Jul 27 16:48:50.838269 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.838250    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy"
Jul 27 16:48:50.838344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.838274    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy"
Jul 27 16:48:50.838403 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.838392    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy"
Jul 27 16:48:50.838437 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.838406    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy"
Jul 27 16:48:50.841119 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.841103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:50.844267 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.844245    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v2ffk\""
Jul 27 16:48:50.857703 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:50.857639    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"]
Jul 27 16:48:51.016627 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.016583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2927c75-8e48-48ae-b162-a70df1d062a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-370eeced-9csvd\" (UID: \"d2927c75-8e48-48ae-b162-a70df1d062a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:51.118117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.118029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2927c75-8e48-48ae-b162-a70df1d062a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-370eeced-9csvd\" (UID: \"d2927c75-8e48-48ae-b162-a70df1d062a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:51.118638 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.118618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2927c75-8e48-48ae-b162-a70df1d062a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-370eeced-9csvd\" (UID: \"d2927c75-8e48-48ae-b162-a70df1d062a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:51.150651 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.150626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:51.173000 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.172969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:51.280931 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.280898    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"]
Jul 27 16:48:51.284452 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:48:51.284426    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2927c75_8e48_48ae_b162_a70df1d062a5.slice/crio-ee4ce7741621f3dda52a66afff101e68c84b52cce03714feee7e136cd7633aa8 WatchSource:0}: Error finding container ee4ce7741621f3dda52a66afff101e68c84b52cce03714feee7e136cd7633aa8: Status 404 returned error can't find the container with id ee4ce7741621f3dda52a66afff101e68c84b52cce03714feee7e136cd7633aa8
Jul 27 16:48:51.403660 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.403625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:51.439648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:51.439615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd" event={"ID":"d2927c75-8e48-48ae-b162-a70df1d062a5","Type":"ContainerStarted","Data":"ee4ce7741621f3dda52a66afff101e68c84b52cce03714feee7e136cd7633aa8"}
Jul 27 16:48:52.445394 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:52.445358    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2927c75-8e48-48ae-b162-a70df1d062a5" containerID="9051d3074175d72798a9524c1ff2d23f6ff3dd623c1dbadd14d43b19fb2af574" exitCode=0
Jul 27 16:48:52.445769 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:52.445447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd" event={"ID":"d2927c75-8e48-48ae-b162-a70df1d062a5","Type":"ContainerDied","Data":"9051d3074175d72798a9524c1ff2d23f6ff3dd623c1dbadd14d43b19fb2af574"}
Jul 27 16:48:53.576557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:53.576530    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:53.742838 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:53.742745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2927c75-8e48-48ae-b162-a70df1d062a5-maas-api-service-ca\") pod \"d2927c75-8e48-48ae-b162-a70df1d062a5\" (UID: \"d2927c75-8e48-48ae-b162-a70df1d062a5\") "
Jul 27 16:48:53.743118 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:53.743093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2927c75-8e48-48ae-b162-a70df1d062a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2927c75-8e48-48ae-b162-a70df1d062a5" (UID: "d2927c75-8e48-48ae-b162-a70df1d062a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:48:53.843779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:53.843741    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2927c75-8e48-48ae-b162-a70df1d062a5-maas-api-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:54.455152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:54.455114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd" event={"ID":"d2927c75-8e48-48ae-b162-a70df1d062a5","Type":"ContainerDied","Data":"ee4ce7741621f3dda52a66afff101e68c84b52cce03714feee7e136cd7633aa8"}
Jul 27 16:48:54.455152 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:54.455141    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"
Jul 27 16:48:54.455380 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:54.455172    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4ce7741621f3dda52a66afff101e68c84b52cce03714feee7e136cd7633aa8"
Jul 27 16:48:54.852976 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:54.852894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:55.390838 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:55.390810    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:57.128739 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:57.128704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:58.012859 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.012831    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"]
Jul 27 16:48:58.013078 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.013055    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2" podUID="5ba1af96-d868-4489-aa91-7997faa14f06" containerName="maas-api" containerID="cri-o://bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805" gracePeriod=30
Jul 27 16:48:58.033487 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.033461    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:58.125108 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.125052    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:58.190918 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.190819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:48:58.267324 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.267242    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:58.381680 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.381649    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba1af96-d868-4489-aa91-7997faa14f06-maas-api-tls\") pod \"5ba1af96-d868-4489-aa91-7997faa14f06\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") "
Jul 27 16:48:58.381868 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.381714    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc88x\" (UniqueName: \"kubernetes.io/projected/5ba1af96-d868-4489-aa91-7997faa14f06-kube-api-access-sc88x\") pod \"5ba1af96-d868-4489-aa91-7997faa14f06\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") "
Jul 27 16:48:58.381868 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.381849    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1af96-d868-4489-aa91-7997faa14f06-openshift-service-ca\") pod \"5ba1af96-d868-4489-aa91-7997faa14f06\" (UID: \"5ba1af96-d868-4489-aa91-7997faa14f06\") "
Jul 27 16:48:58.382194 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.382165    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba1af96-d868-4489-aa91-7997faa14f06-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ba1af96-d868-4489-aa91-7997faa14f06" (UID: "5ba1af96-d868-4489-aa91-7997faa14f06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:48:58.383704 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.383673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba1af96-d868-4489-aa91-7997faa14f06-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ba1af96-d868-4489-aa91-7997faa14f06" (UID: "5ba1af96-d868-4489-aa91-7997faa14f06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:48:58.383825 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.383804    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba1af96-d868-4489-aa91-7997faa14f06-kube-api-access-sc88x" (OuterVolumeSpecName: "kube-api-access-sc88x") pod "5ba1af96-d868-4489-aa91-7997faa14f06" (UID: "5ba1af96-d868-4489-aa91-7997faa14f06"). InnerVolumeSpecName "kube-api-access-sc88x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:48:58.474445 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.474411    2559 generic.go:356] "Generic (PLEG): container finished" podID="5ba1af96-d868-4489-aa91-7997faa14f06" containerID="bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805" exitCode=0
Jul 27 16:48:58.474610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.474456    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"
Jul 27 16:48:58.474610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.474497    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2" event={"ID":"5ba1af96-d868-4489-aa91-7997faa14f06","Type":"ContainerDied","Data":"bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805"}
Jul 27 16:48:58.474610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.474546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2" event={"ID":"5ba1af96-d868-4489-aa91-7997faa14f06","Type":"ContainerDied","Data":"6f4df0daf80c11220818f6e94d4e2a45a1410afb5b21c317b2f8c97ec6fe4d7e"}
Jul 27 16:48:58.474610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.474568    2559 scope.go:117] "RemoveContainer" containerID="bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805"
Jul 27 16:48:58.482574 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.482552    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc88x\" (UniqueName: \"kubernetes.io/projected/5ba1af96-d868-4489-aa91-7997faa14f06-kube-api-access-sc88x\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:58.482667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.482578    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba1af96-d868-4489-aa91-7997faa14f06-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:58.482667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.482595    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ba1af96-d868-4489-aa91-7997faa14f06-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:48:58.482915 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.482898    2559 scope.go:117] "RemoveContainer" containerID="bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805"
Jul 27 16:48:58.483118 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:48:58.483103    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805\": container with ID starting with bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805 not found: ID does not exist" containerID="bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805"
Jul 27 16:48:58.483167 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.483126    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805"} err="failed to get container status \"bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805\": rpc error: code = NotFound desc = could not find container \"bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805\": container with ID starting with bdf81c0a94589e7345ed47ef2f4dba3a03d4ee09ac19408b64aba83c4c43d805 not found: ID does not exist"
Jul 27 16:48:58.495445 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.495423    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"]
Jul 27 16:48:58.499386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.499367    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-370eeced-8656bfdc67-44nd2"]
Jul 27 16:48:58.735372 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:48:58.735340    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba1af96-d868-4489-aa91-7997faa14f06" path="/var/lib/kubelet/pods/5ba1af96-d868-4489-aa91-7997faa14f06/volumes"
Jul 27 16:49:02.652207 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.652173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:02.664981 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.664946    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"]
Jul 27 16:49:02.668009 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.667984    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-370eeced-9csvd"]
Jul 27 16:49:02.735263 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.735231    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2927c75-8e48-48ae-b162-a70df1d062a5" path="/var/lib/kubelet/pods/d2927c75-8e48-48ae-b162-a70df1d062a5/volumes"
Jul 27 16:49:02.817447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.817412    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"]
Jul 27 16:49:02.817700 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.817674    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy" containerID="cri-o://f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8" gracePeriod=30
Jul 27 16:49:02.828504 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:02.828473    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:03.061474 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:03.061396    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:03.546527 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:03.546494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:03.656296 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:03.656252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:04.423331 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:04.423300    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:05.246327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:05.246274    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:07.109615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:07.109585    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:08.018151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.018121    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:08.071858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.071834    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:49:08.152406 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152380    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-certs\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-socket\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152454    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e448cf-02e1-47b2-bc79-4e226f7756c0-istiod-ca-cert\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktvpd\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-kube-api-access-ktvpd\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152502    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-podinfo\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-credential-socket\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152536    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-data\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152553    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-envoy\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152574    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-token\") pod \"20e448cf-02e1-47b2-bc79-4e226f7756c0\" (UID: \"20e448cf-02e1-47b2-bc79-4e226f7756c0\") "
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152688    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152712    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152782    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:49:08.152834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152836    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.153318 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152849    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.153318 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.152905    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e448cf-02e1-47b2-bc79-4e226f7756c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:49:08.153946 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.153918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:49:08.154680 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.154650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:49:08.154825 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.154808    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:49:08.155022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.155002    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-kube-api-access-ktvpd" (OuterVolumeSpecName: "kube-api-access-ktvpd") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "kube-api-access-ktvpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:49:08.155022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.154999    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20e448cf-02e1-47b2-bc79-4e226f7756c0" (UID: "20e448cf-02e1-47b2-bc79-4e226f7756c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:49:08.253420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253385    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20e448cf-02e1-47b2-bc79-4e226f7756c0-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.253420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253414    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvpd\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-kube-api-access-ktvpd\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.253420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253428    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.253649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253440    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.253649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253452    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.253649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253465    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.253649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.253475    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20e448cf-02e1-47b2-bc79-4e226f7756c0-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:49:08.519479 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.519401    2559 generic.go:356] "Generic (PLEG): container finished" podID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerID="f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8" exitCode=0
Jul 27 16:49:08.519479 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.519454    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"
Jul 27 16:49:08.519652 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.519452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" event={"ID":"20e448cf-02e1-47b2-bc79-4e226f7756c0","Type":"ContainerDied","Data":"f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8"}
Jul 27 16:49:08.519652 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.519576    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh" event={"ID":"20e448cf-02e1-47b2-bc79-4e226f7756c0","Type":"ContainerDied","Data":"186d51c7ded412ff55db732831f0e408059fa91f6b76d08f7cdb1a367c80afa8"}
Jul 27 16:49:08.519652 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.519605    2559 scope.go:117] "RemoveContainer" containerID="f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8"
Jul 27 16:49:08.528393 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.528379    2559 scope.go:117] "RemoveContainer" containerID="f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"
Jul 27 16:49:08.535494 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.535478    2559 scope.go:117] "RemoveContainer" containerID="f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8"
Jul 27 16:49:08.535717 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:49:08.535703    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8\": container with ID starting with f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8 not found: ID does not exist" containerID="f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8"
Jul 27 16:49:08.535769 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.535726    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8"} err="failed to get container status \"f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8\": rpc error: code = NotFound desc = could not find container \"f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8\": container with ID starting with f9bbcee0a10c2e75883fb665af1aeedf92898e63ee7a3888f05762144e2a5bb8 not found: ID does not exist"
Jul 27 16:49:08.535769 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.535741    2559 scope.go:117] "RemoveContainer" containerID="f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"
Jul 27 16:49:08.535937 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:49:08.535916    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd\": container with ID starting with f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd not found: ID does not exist" containerID="f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"
Jul 27 16:49:08.535989 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.535947    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd"} err="failed to get container status \"f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd\": rpc error: code = NotFound desc = could not find container \"f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd\": container with ID starting with f04bbbfc344f61f98989d0d29e790ac304cf3f06c03898ddbcbd4e0b16306edd not found: ID does not exist"
Jul 27 16:49:08.541448 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.541425    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"]
Jul 27 16:49:08.548388 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.548368    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-370eeced-openshift-default-747764785b-lbdbh"]
Jul 27 16:49:08.735688 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:08.735653    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" path="/var/lib/kubelet/pods/20e448cf-02e1-47b2-bc79-4e226f7756c0/volumes"
Jul 27 16:49:13.754523 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:13.754454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:40.894350 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:40.894308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:43.806269 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:43.806235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:46.841895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:46.841863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:47.416748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:47.416712    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:49.698753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.698721    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-45f9r"]
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.698992    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2927c75-8e48-48ae-b162-a70df1d062a5" containerName="revoke-keys"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699004    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2927c75-8e48-48ae-b162-a70df1d062a5" containerName="revoke-keys"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699014    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba1af96-d868-4489-aa91-7997faa14f06" containerName="maas-api"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699020    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba1af96-d868-4489-aa91-7997faa14f06" containerName="maas-api"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699030    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699035    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699043    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699048    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699055    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699060    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f244cc8d-d03c-449a-aa63-dde96dbfc383" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699110    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba1af96-d868-4489-aa91-7997faa14f06" containerName="maas-api"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699118    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2927c75-8e48-48ae-b162-a70df1d062a5" containerName="revoke-keys"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699124    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy"
Jul 27 16:49:49.699129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.699129    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e448cf-02e1-47b2-bc79-4e226f7756c0" containerName="istio-proxy"
Jul 27 16:49:49.701960 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.701942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.704839 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.704818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Jul 27 16:49:49.799429 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.799391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/a19ca411-200c-4a2e-99f4-106836a536fa-ready\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.799599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.799438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a19ca411-200c-4a2e-99f4-106836a536fa-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.799599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.799534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9ng\" (UniqueName: \"kubernetes.io/projected/a19ca411-200c-4a2e-99f4-106836a536fa-kube-api-access-vm9ng\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.799599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.799568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a19ca411-200c-4a2e-99f4-106836a536fa-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.846426 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.846398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:49.900533 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.900505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/a19ca411-200c-4a2e-99f4-106836a536fa-ready\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.900646 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.900544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a19ca411-200c-4a2e-99f4-106836a536fa-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.900707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.900671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9ng\" (UniqueName: \"kubernetes.io/projected/a19ca411-200c-4a2e-99f4-106836a536fa-kube-api-access-vm9ng\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.900760 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.900711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a19ca411-200c-4a2e-99f4-106836a536fa-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.900819 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.900794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a19ca411-200c-4a2e-99f4-106836a536fa-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.900919 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.900899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/a19ca411-200c-4a2e-99f4-106836a536fa-ready\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.901023 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.901007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a19ca411-200c-4a2e-99f4-106836a536fa-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.910519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.910501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9ng\" (UniqueName: \"kubernetes.io/projected/a19ca411-200c-4a2e-99f4-106836a536fa-kube-api-access-vm9ng\") pod \"cni-sysctl-allowlist-ds-45f9r\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") " pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:49.943368 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:49.943343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:50.010656 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.010583    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:50.021407 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:49:50.021383    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19ca411_200c_4a2e_99f4_106836a536fa.slice/crio-332aef998c403f847abfd1f778e64f7747e557cc163031ebc4f0033d93d834f8 WatchSource:0}: Error finding container 332aef998c403f847abfd1f778e64f7747e557cc163031ebc4f0033d93d834f8: Status 404 returned error can't find the container with id 332aef998c403f847abfd1f778e64f7747e557cc163031ebc4f0033d93d834f8
Jul 27 16:49:50.040896 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.040866    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:50.697188 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.697156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" event={"ID":"a19ca411-200c-4a2e-99f4-106836a536fa","Type":"ContainerStarted","Data":"41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b"}
Jul 27 16:49:50.697386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.697205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" event={"ID":"a19ca411-200c-4a2e-99f4-106836a536fa","Type":"ContainerStarted","Data":"332aef998c403f847abfd1f778e64f7747e557cc163031ebc4f0033d93d834f8"}
Jul 27 16:49:50.697386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.697358    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:50.718235 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.718181    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" podStartSLOduration=1.718157013 podStartE2EDuration="1.718157013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:49:50.717414474 +0000 UTC m=+1990.618554921" watchObservedRunningTime="2026-07-27 16:49:50.718157013 +0000 UTC m=+1990.619297459"
Jul 27 16:49:50.809543 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.809513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:50.911130 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:50.911100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:51.385928 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:51.385895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:51.697682 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:51.697645    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:51.710229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:51.710208    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:49:52.347180 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:52.347143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:52.694179 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:52.694120    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-45f9r"]
Jul 27 16:49:52.955693 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:52.955613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:49:53.709360 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:49:53.709314    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" gracePeriod=30
Jul 27 16:50:00.752145 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:00.752109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:00.883273 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:00.883240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:00.983269 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:00.983244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:01.250166 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:01.250129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:01.349500 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:01.349463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:01.702364 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:01.702326    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:01.703388 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:01.703351    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:01.704201 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:01.704170    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:01.704271 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:01.704213    2559 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 27 16:50:04.007981 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:04.007940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:04.108964 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:04.108932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:04.208497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:04.208468    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:11.702159 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:11.702118    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:11.703037 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:11.703010    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:11.703827 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:11.703795    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:11.703905 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:11.703836    2559 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 27 16:50:13.017052 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:13.017018    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:13.128435 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:13.128395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:13.223702 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:13.223666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:13.323641 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:13.323561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:13.770950 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:13.770918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:21.702040 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:21.702000    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:21.702871 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:21.702841    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:21.703671 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:21.703641    2559 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:21.703731 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:21.703684    2559 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 27 16:50:23.792395 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.792371    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-45f9r_a19ca411-200c-4a2e-99f4-106836a536fa/kube-multus-additional-cni-plugins/0.log"
Jul 27 16:50:23.794646 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.794628    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:50:23.835877 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.835851    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-45f9r_a19ca411-200c-4a2e-99f4-106836a536fa/kube-multus-additional-cni-plugins/0.log"
Jul 27 16:50:23.836038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.835892    2559 generic.go:356] "Generic (PLEG): container finished" podID="a19ca411-200c-4a2e-99f4-106836a536fa" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b" exitCode=137
Jul 27 16:50:23.836038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.835941    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r"
Jul 27 16:50:23.836038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.835963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" event={"ID":"a19ca411-200c-4a2e-99f4-106836a536fa","Type":"ContainerDied","Data":"41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b"}
Jul 27 16:50:23.836038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.836010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-45f9r" event={"ID":"a19ca411-200c-4a2e-99f4-106836a536fa","Type":"ContainerDied","Data":"332aef998c403f847abfd1f778e64f7747e557cc163031ebc4f0033d93d834f8"}
Jul 27 16:50:23.836038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.836036    2559 scope.go:117] "RemoveContainer" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b"
Jul 27 16:50:23.844061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.844044    2559 scope.go:117] "RemoveContainer" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b"
Jul 27 16:50:23.844313 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:50:23.844272    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b\": container with ID starting with 41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b not found: ID does not exist" containerID="41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b"
Jul 27 16:50:23.844379 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.844325    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b"} err="failed to get container status \"41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b\": rpc error: code = NotFound desc = could not find container \"41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b\": container with ID starting with 41cd733fecb9de870c0eb77c5cb031445f38158150b040f93b3c36fcdb22257b not found: ID does not exist"
Jul 27 16:50:23.881714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.881687    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/a19ca411-200c-4a2e-99f4-106836a536fa-ready\") pod \"a19ca411-200c-4a2e-99f4-106836a536fa\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") "
Jul 27 16:50:23.881851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.881759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm9ng\" (UniqueName: \"kubernetes.io/projected/a19ca411-200c-4a2e-99f4-106836a536fa-kube-api-access-vm9ng\") pod \"a19ca411-200c-4a2e-99f4-106836a536fa\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") "
Jul 27 16:50:23.881851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.881788    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a19ca411-200c-4a2e-99f4-106836a536fa-cni-sysctl-allowlist\") pod \"a19ca411-200c-4a2e-99f4-106836a536fa\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") "
Jul 27 16:50:23.881851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.881829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a19ca411-200c-4a2e-99f4-106836a536fa-tuning-conf-dir\") pod \"a19ca411-200c-4a2e-99f4-106836a536fa\" (UID: \"a19ca411-200c-4a2e-99f4-106836a536fa\") "
Jul 27 16:50:23.882003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.881954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/a19ca411-200c-4a2e-99f4-106836a536fa-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "a19ca411-200c-4a2e-99f4-106836a536fa" (UID: "a19ca411-200c-4a2e-99f4-106836a536fa"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 27 16:50:23.882003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.881966    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19ca411-200c-4a2e-99f4-106836a536fa-ready" (OuterVolumeSpecName: "ready") pod "a19ca411-200c-4a2e-99f4-106836a536fa" (UID: "a19ca411-200c-4a2e-99f4-106836a536fa"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:50:23.882133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.882107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ca411-200c-4a2e-99f4-106836a536fa-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "a19ca411-200c-4a2e-99f4-106836a536fa" (UID: "a19ca411-200c-4a2e-99f4-106836a536fa"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:50:23.882133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.882113    2559 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/a19ca411-200c-4a2e-99f4-106836a536fa-ready\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:50:23.882267 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.882153    2559 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a19ca411-200c-4a2e-99f4-106836a536fa-tuning-conf-dir\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:50:23.883774 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.883756    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19ca411-200c-4a2e-99f4-106836a536fa-kube-api-access-vm9ng" (OuterVolumeSpecName: "kube-api-access-vm9ng") pod "a19ca411-200c-4a2e-99f4-106836a536fa" (UID: "a19ca411-200c-4a2e-99f4-106836a536fa"). InnerVolumeSpecName "kube-api-access-vm9ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:50:23.983470 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.983389    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm9ng\" (UniqueName: \"kubernetes.io/projected/a19ca411-200c-4a2e-99f4-106836a536fa-kube-api-access-vm9ng\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:50:23.983470 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:23.983421    2559 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a19ca411-200c-4a2e-99f4-106836a536fa-cni-sysctl-allowlist\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:50:24.162936 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:24.162905    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-45f9r"]
Jul 27 16:50:24.166493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:24.166468    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-45f9r"]
Jul 27 16:50:24.735503 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:24.735471    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" path="/var/lib/kubelet/pods/a19ca411-200c-4a2e-99f4-106836a536fa/volumes"
Jul 27 16:50:32.836837 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:32.836807    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:39.748132 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:39.748091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:46.337465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:46.337430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:46.439251 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:46.439217    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:50:46.543347 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:50:46.543297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:02.440156 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:02.440122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:08.868269 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:08.868238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:08.975952 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:08.975922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:09.588556 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:09.588523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:19.651265 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:19.646764    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:20.516006 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:20.515976    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:27.568887 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:27.568856    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:27.679919 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:27.679888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:27.804123 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:27.804085    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:27.903294 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:27.903257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:28.702941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:28.702906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:28.819622 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:28.819588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:30.700386 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:30.700348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:36.654399 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:36.654362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:36.757204 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:36.757166    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:37.391304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:37.391248    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:44.994410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:44.994371    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:45.094314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.094260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:45.363817 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.363741    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"]
Jul 27 16:51:45.364037 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.364025    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins"
Jul 27 16:51:45.364078 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.364038    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins"
Jul 27 16:51:45.364112 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.364090    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19ca411-200c-4a2e-99f4-106836a536fa" containerName="kube-multus-additional-cni-plugins"
Jul 27 16:51:45.366911 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.366885    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.369903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.369881    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fee06a22-openshift-default-dockercfg-qb98l\""
Jul 27 16:51:45.391168 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.391145    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"]
Jul 27 16:51:45.417022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.416988    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:45.464762 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464733    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-certs\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.464910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-socket\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.464910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be63c8e4-d1e5-46f0-b307-437646b71986-istio-podinfo\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.464910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-data\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.464910 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be63c8e4-d1e5-46f0-b307-437646b71986-istiod-ca-cert\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.465056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-istio-token\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.465056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4lk\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-kube-api-access-5t4lk\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.465056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.464962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-credential-socket\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.465056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.465002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-envoy\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565756 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-certs\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-socket\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be63c8e4-d1e5-46f0-b307-437646b71986-istio-podinfo\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-data\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be63c8e4-d1e5-46f0-b307-437646b71986-istiod-ca-cert\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-istio-token\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4lk\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-kube-api-access-5t4lk\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.565929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-credential-socket\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.566366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.565985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-envoy\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.566366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.566134    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-certs\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.566366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.566331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-data\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.566486 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.566380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-socket\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.566486 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.566473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-credential-socket\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.566582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.566564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be63c8e4-d1e5-46f0-b307-437646b71986-istiod-ca-cert\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.568081 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.568062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-envoy\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.568181 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.568156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be63c8e4-d1e5-46f0-b307-437646b71986-istio-podinfo\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.577250 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.577226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-istio-token\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.577555 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.577537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4lk\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-kube-api-access-5t4lk\") pod \"e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") " pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.657436 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.657403    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:45.677575 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.677548    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:45.807827 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.807800    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"]
Jul 27 16:51:45.809910 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:51:45.809884    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe63c8e4_d1e5_46f0_b307_437646b71986.slice/crio-35a9ba0815d22f545d7e767677254b1419bcd5371233986211006f93906d74e8 WatchSource:0}: Error finding container 35a9ba0815d22f545d7e767677254b1419bcd5371233986211006f93906d74e8: Status 404 returned error can't find the container with id 35a9ba0815d22f545d7e767677254b1419bcd5371233986211006f93906d74e8
Jul 27 16:51:45.811963 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.811934    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:51:45.812057 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.811993    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:51:45.812057 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:45.812022    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:51:46.163033 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:46.162986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" event={"ID":"be63c8e4-d1e5-46f0-b307-437646b71986","Type":"ContainerStarted","Data":"1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"}
Jul 27 16:51:46.163464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:46.163050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" event={"ID":"be63c8e4-d1e5-46f0-b307-437646b71986","Type":"ContainerStarted","Data":"35a9ba0815d22f545d7e767677254b1419bcd5371233986211006f93906d74e8"}
Jul 27 16:51:46.187529 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:46.187482    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podStartSLOduration=1.187467319 podStartE2EDuration="1.187467319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:51:46.185899986 +0000 UTC m=+2106.087040432" watchObservedRunningTime="2026-07-27 16:51:46.187467319 +0000 UTC m=+2106.088607766"
Jul 27 16:51:46.678230 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:46.678192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:46.679509 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:46.679484    2559 patch_prober.go:28] interesting pod/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 27 16:51:46.679632 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:46.679532    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 27 16:51:47.681590 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:47.681552    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:48.172186 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.172142    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:48.173001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.172983    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:51:48.195133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.195102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:48.301891 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.301854    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:48.316087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.316054    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"]
Jul 27 16:51:48.319456 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.319440    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.323025 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.322996    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:51:48.323133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.323044    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:51:48.323467 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.323452    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:51:48.323580 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.323564    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fee06a22-serving-cert\""
Jul 27 16:51:48.332372 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.332348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"]
Jul 27 16:51:48.404792 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.404760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:48.491100 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.491012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg952\" (UniqueName: \"kubernetes.io/projected/864aeb5f-09d8-479d-9f63-23fc78bbbf81-kube-api-access-rg952\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.491100 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.491078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864aeb5f-09d8-479d-9f63-23fc78bbbf81-maas-api-tls\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.491266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.491113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864aeb5f-09d8-479d-9f63-23fc78bbbf81-openshift-service-ca\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.502000 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.501977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:48.591658 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.591625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864aeb5f-09d8-479d-9f63-23fc78bbbf81-maas-api-tls\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.591793 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.591662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864aeb5f-09d8-479d-9f63-23fc78bbbf81-openshift-service-ca\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.591793 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.591714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg952\" (UniqueName: \"kubernetes.io/projected/864aeb5f-09d8-479d-9f63-23fc78bbbf81-kube-api-access-rg952\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.592432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.592414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864aeb5f-09d8-479d-9f63-23fc78bbbf81-openshift-service-ca\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.593971 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.593945    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864aeb5f-09d8-479d-9f63-23fc78bbbf81-maas-api-tls\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.600871 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.600848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg952\" (UniqueName: \"kubernetes.io/projected/864aeb5f-09d8-479d-9f63-23fc78bbbf81-kube-api-access-rg952\") pod \"maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.629852 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.629824    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:48.761487 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.761462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"]
Jul 27 16:51:48.763777 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:51:48.763744    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864aeb5f_09d8_479d_9f63_23fc78bbbf81.slice/crio-6514141c192640a85a6ca50e904e1a56354d8935a43145faf1a00b705c5f0ecb WatchSource:0}: Error finding container 6514141c192640a85a6ca50e904e1a56354d8935a43145faf1a00b705c5f0ecb: Status 404 returned error can't find the container with id 6514141c192640a85a6ca50e904e1a56354d8935a43145faf1a00b705c5f0ecb
Jul 27 16:51:48.765173 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.765157    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:51:48.957656 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:48.957626    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:49.063487 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.063405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:49.068797 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.068771    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"]
Jul 27 16:51:49.073420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.073404    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"]
Jul 27 16:51:49.073540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.073527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.076404 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.076381    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.076612 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.076597    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fee06a22\""
Jul 27 16:51:49.082601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.082579    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"]
Jul 27 16:51:49.086410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.086389    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"]
Jul 27 16:51:49.177420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.177388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt" event={"ID":"864aeb5f-09d8-479d-9f63-23fc78bbbf81","Type":"ContainerStarted","Data":"6514141c192640a85a6ca50e904e1a56354d8935a43145faf1a00b705c5f0ecb"}
Jul 27 16:51:49.198064 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.198027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsjc\" (UniqueName: \"kubernetes.io/projected/1d89df17-68ce-46c6-8d9c-3de3b489acb1-kube-api-access-snsjc\") pod \"payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.198238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.198126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58pzv\" (UniqueName: \"kubernetes.io/projected/872aa7f7-e836-40b6-8db1-2b87916fa470-kube-api-access-58pzv\") pod \"payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") " pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.198238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.198181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d89df17-68ce-46c6-8d9c-3de3b489acb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.198238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.198220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872aa7f7-e836-40b6-8db1-2b87916fa470-plugins-config-volume\") pod \"payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") " pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.299121 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.299096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsjc\" (UniqueName: \"kubernetes.io/projected/1d89df17-68ce-46c6-8d9c-3de3b489acb1-kube-api-access-snsjc\") pod \"payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.299224 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.299161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58pzv\" (UniqueName: \"kubernetes.io/projected/872aa7f7-e836-40b6-8db1-2b87916fa470-kube-api-access-58pzv\") pod \"payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") " pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.299291 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.299256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d89df17-68ce-46c6-8d9c-3de3b489acb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.299352 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.299323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872aa7f7-e836-40b6-8db1-2b87916fa470-plugins-config-volume\") pod \"payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") " pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.299959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.299941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872aa7f7-e836-40b6-8db1-2b87916fa470-plugins-config-volume\") pod \"payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") " pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.300066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.300048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d89df17-68ce-46c6-8d9c-3de3b489acb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.309086 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.309063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58pzv\" (UniqueName: \"kubernetes.io/projected/872aa7f7-e836-40b6-8db1-2b87916fa470-kube-api-access-58pzv\") pod \"payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") " pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.309169 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.309144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsjc\" (UniqueName: \"kubernetes.io/projected/1d89df17-68ce-46c6-8d9c-3de3b489acb1-kube-api-access-snsjc\") pod \"payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.386154 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.386075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:49.392356 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.392330    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:49.551888 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.551864    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"]
Jul 27 16:51:49.554141 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:51:49.554114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872aa7f7_e836_40b6_8db1_2b87916fa470.slice/crio-4c258405cb2cf7f4587298a989a1414309f94b4df2ad3853a0472f0d895d35b4 WatchSource:0}: Error finding container 4c258405cb2cf7f4587298a989a1414309f94b4df2ad3853a0472f0d895d35b4: Status 404 returned error can't find the container with id 4c258405cb2cf7f4587298a989a1414309f94b4df2ad3853a0472f0d895d35b4
Jul 27 16:51:49.733043 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:49.733020    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"]
Jul 27 16:51:49.735137 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:51:49.735102    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d89df17_68ce_46c6_8d9c_3de3b489acb1.slice/crio-786d9a7a7b8c3054b80cb3a5186cec4a25d6fa1cca41f89c198fe4770f4baaed WatchSource:0}: Error finding container 786d9a7a7b8c3054b80cb3a5186cec4a25d6fa1cca41f89c198fe4770f4baaed: Status 404 returned error can't find the container with id 786d9a7a7b8c3054b80cb3a5186cec4a25d6fa1cca41f89c198fe4770f4baaed
Jul 27 16:51:50.184039 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.184003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt" event={"ID":"864aeb5f-09d8-479d-9f63-23fc78bbbf81","Type":"ContainerStarted","Data":"9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0"}
Jul 27 16:51:50.184521 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.184116    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:51:50.185640 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.185614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v" event={"ID":"872aa7f7-e836-40b6-8db1-2b87916fa470","Type":"ContainerStarted","Data":"df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b"}
Jul 27 16:51:50.185777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.185659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v" event={"ID":"872aa7f7-e836-40b6-8db1-2b87916fa470","Type":"ContainerStarted","Data":"4c258405cb2cf7f4587298a989a1414309f94b4df2ad3853a0472f0d895d35b4"}
Jul 27 16:51:50.185777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.185711    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:51:50.187002 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.186979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf" event={"ID":"1d89df17-68ce-46c6-8d9c-3de3b489acb1","Type":"ContainerStarted","Data":"d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c"}
Jul 27 16:51:50.187083 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.187024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf" event={"ID":"1d89df17-68ce-46c6-8d9c-3de3b489acb1","Type":"ContainerStarted","Data":"786d9a7a7b8c3054b80cb3a5186cec4a25d6fa1cca41f89c198fe4770f4baaed"}
Jul 27 16:51:50.187167 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.187153    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:51:50.204700 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.204656    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt" podStartSLOduration=1.726446618 podStartE2EDuration="2.204642275s" totalImagesPullingTime="478.195657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:51:50.203159581 +0000 UTC m=+2110.104300033" watchObservedRunningTime="2026-07-27 16:51:50.204642275 +0000 UTC m=+2110.105782721"
Jul 27 16:51:50.221557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.221519    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v" podStartSLOduration=1.221506391 podStartE2EDuration="1.221506391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:51:50.220701935 +0000 UTC m=+2110.121842381" watchObservedRunningTime="2026-07-27 16:51:50.221506391 +0000 UTC m=+2110.122646838"
Jul 27 16:51:50.240733 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:50.240694    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf" podStartSLOduration=1.240678116 podStartE2EDuration="1.240678116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:51:50.240359049 +0000 UTC m=+2110.141499500" watchObservedRunningTime="2026-07-27 16:51:50.240678116 +0000 UTC m=+2110.141818563"
Jul 27 16:51:53.942121 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:53.942085    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:54.038876 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:54.038845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:51:56.201939 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:51:56.201907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:52:00.049327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:00.049292    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:00.154443 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:00.154415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:01.198498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.198411    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:52:01.198969 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.198588    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:52:01.241493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.241458    2559 generic.go:356] "Generic (PLEG): container finished" podID="be63c8e4-d1e5-46f0-b307-437646b71986" containerID="1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f" exitCode=0
Jul 27 16:52:01.241670 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.241540    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" event={"ID":"be63c8e4-d1e5-46f0-b307-437646b71986","Type":"ContainerDied","Data":"1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"}
Jul 27 16:52:01.241942 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.241925    2559 scope.go:117] "RemoveContainer" containerID="1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"
Jul 27 16:52:01.243049 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.243023    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:01.243128 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.243084    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:01.243128 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:01.243116    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:02.250365 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:02.250330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" event={"ID":"be63c8e4-d1e5-46f0-b307-437646b71986","Type":"ContainerStarted","Data":"4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64"}
Jul 27 16:52:02.678604 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:02.678567    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:52:02.679899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:02.679871    2559 patch_prober.go:28] interesting pod/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 27 16:52:02.680031 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:02.679930    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 27 16:52:03.172480 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:03.172444    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:52:03.679040 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:03.678998    2559 patch_prober.go:28] interesting pod/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 27 16:52:03.679468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:03.679085    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 27 16:52:04.678519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:04.678481    2559 patch_prober.go:28] interesting pod/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 27 16:52:04.678708 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:04.678533    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 27 16:52:05.678828 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:05.678778    2559 patch_prober.go:28] interesting pod/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Jul 27 16:52:05.679212 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:05.678844    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Jul 27 16:52:06.681648 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:06.681613    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:52:06.682551 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:06.682532    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:52:10.811937 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.811905    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"]
Jul 27 16:52:10.812410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.812129    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt" podUID="864aeb5f-09d8-479d-9f63-23fc78bbbf81" containerName="maas-api" containerID="cri-o://9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0" gracePeriod=30
Jul 27 16:52:10.845151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.845116    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:10.872428 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.872396    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"]
Jul 27 16:52:10.872667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.872640    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v" podUID="872aa7f7-e836-40b6-8db1-2b87916fa470" containerName="payload-processing" containerID="cri-o://df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b" gracePeriod=30
Jul 27 16:52:10.885189 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.885158    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"]
Jul 27 16:52:10.885459 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.885421    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf" podUID="1d89df17-68ce-46c6-8d9c-3de3b489acb1" containerName="payload-pre-processing" containerID="cri-o://d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c" gracePeriod=30
Jul 27 16:52:10.922997 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.922966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:10.987802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:10.987772    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:11.152297 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.152249    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:52:11.155000 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.154976    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:52:11.172111 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.172088    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.178412    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872aa7f7-e836-40b6-8db1-2b87916fa470-plugins-config-volume\") pod \"872aa7f7-e836-40b6-8db1-2b87916fa470\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") "
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.178469    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864aeb5f-09d8-479d-9f63-23fc78bbbf81-maas-api-tls\") pod \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") "
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.178496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58pzv\" (UniqueName: \"kubernetes.io/projected/872aa7f7-e836-40b6-8db1-2b87916fa470-kube-api-access-58pzv\") pod \"872aa7f7-e836-40b6-8db1-2b87916fa470\" (UID: \"872aa7f7-e836-40b6-8db1-2b87916fa470\") "
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.178570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg952\" (UniqueName: \"kubernetes.io/projected/864aeb5f-09d8-479d-9f63-23fc78bbbf81-kube-api-access-rg952\") pod \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") "
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.178598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864aeb5f-09d8-479d-9f63-23fc78bbbf81-openshift-service-ca\") pod \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\" (UID: \"864aeb5f-09d8-479d-9f63-23fc78bbbf81\") "
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.178958    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872aa7f7-e836-40b6-8db1-2b87916fa470-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "872aa7f7-e836-40b6-8db1-2b87916fa470" (UID: "872aa7f7-e836-40b6-8db1-2b87916fa470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:52:11.179661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.179113    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864aeb5f-09d8-479d-9f63-23fc78bbbf81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "864aeb5f-09d8-479d-9f63-23fc78bbbf81" (UID: "864aeb5f-09d8-479d-9f63-23fc78bbbf81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:52:11.181258 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.181236    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872aa7f7-e836-40b6-8db1-2b87916fa470-kube-api-access-58pzv" (OuterVolumeSpecName: "kube-api-access-58pzv") pod "872aa7f7-e836-40b6-8db1-2b87916fa470" (UID: "872aa7f7-e836-40b6-8db1-2b87916fa470"). InnerVolumeSpecName "kube-api-access-58pzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:52:11.181409 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.181337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864aeb5f-09d8-479d-9f63-23fc78bbbf81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "864aeb5f-09d8-479d-9f63-23fc78bbbf81" (UID: "864aeb5f-09d8-479d-9f63-23fc78bbbf81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:52:11.181583 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.181555    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864aeb5f-09d8-479d-9f63-23fc78bbbf81-kube-api-access-rg952" (OuterVolumeSpecName: "kube-api-access-rg952") pod "864aeb5f-09d8-479d-9f63-23fc78bbbf81" (UID: "864aeb5f-09d8-479d-9f63-23fc78bbbf81"). InnerVolumeSpecName "kube-api-access-rg952". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:52:11.280048 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snsjc\" (UniqueName: \"kubernetes.io/projected/1d89df17-68ce-46c6-8d9c-3de3b489acb1-kube-api-access-snsjc\") pod \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") "
Jul 27 16:52:11.280222 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d89df17-68ce-46c6-8d9c-3de3b489acb1-plugins-config-volume\") pod \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\" (UID: \"1d89df17-68ce-46c6-8d9c-3de3b489acb1\") "
Jul 27 16:52:11.280274 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280256    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/872aa7f7-e836-40b6-8db1-2b87916fa470-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:11.280274 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280267    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864aeb5f-09d8-479d-9f63-23fc78bbbf81-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:11.280370 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280275    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58pzv\" (UniqueName: \"kubernetes.io/projected/872aa7f7-e836-40b6-8db1-2b87916fa470-kube-api-access-58pzv\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:11.280370 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280315    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg952\" (UniqueName: \"kubernetes.io/projected/864aeb5f-09d8-479d-9f63-23fc78bbbf81-kube-api-access-rg952\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:11.280370 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280328    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864aeb5f-09d8-479d-9f63-23fc78bbbf81-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:11.280494 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.280472    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d89df17-68ce-46c6-8d9c-3de3b489acb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d89df17-68ce-46c6-8d9c-3de3b489acb1" (UID: "1d89df17-68ce-46c6-8d9c-3de3b489acb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:52:11.282087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.282063    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d89df17-68ce-46c6-8d9c-3de3b489acb1-kube-api-access-snsjc" (OuterVolumeSpecName: "kube-api-access-snsjc") pod "1d89df17-68ce-46c6-8d9c-3de3b489acb1" (UID: "1d89df17-68ce-46c6-8d9c-3de3b489acb1"). InnerVolumeSpecName "kube-api-access-snsjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:52:11.294992 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.294961    2559 generic.go:356] "Generic (PLEG): container finished" podID="864aeb5f-09d8-479d-9f63-23fc78bbbf81" containerID="9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0" exitCode=0
Jul 27 16:52:11.295151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.295006    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"
Jul 27 16:52:11.295151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.295032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt" event={"ID":"864aeb5f-09d8-479d-9f63-23fc78bbbf81","Type":"ContainerDied","Data":"9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0"}
Jul 27 16:52:11.295151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.295069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt" event={"ID":"864aeb5f-09d8-479d-9f63-23fc78bbbf81","Type":"ContainerDied","Data":"6514141c192640a85a6ca50e904e1a56354d8935a43145faf1a00b705c5f0ecb"}
Jul 27 16:52:11.295151 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.295090    2559 scope.go:117] "RemoveContainer" containerID="9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0"
Jul 27 16:52:11.296264 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.296243    2559 generic.go:356] "Generic (PLEG): container finished" podID="872aa7f7-e836-40b6-8db1-2b87916fa470" containerID="df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b" exitCode=0
Jul 27 16:52:11.296366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.296308    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"
Jul 27 16:52:11.296366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.296317    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v" event={"ID":"872aa7f7-e836-40b6-8db1-2b87916fa470","Type":"ContainerDied","Data":"df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b"}
Jul 27 16:52:11.296468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.296377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v" event={"ID":"872aa7f7-e836-40b6-8db1-2b87916fa470","Type":"ContainerDied","Data":"4c258405cb2cf7f4587298a989a1414309f94b4df2ad3853a0472f0d895d35b4"}
Jul 27 16:52:11.297677 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.297656    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d89df17-68ce-46c6-8d9c-3de3b489acb1" containerID="d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c" exitCode=0
Jul 27 16:52:11.297801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.297682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf" event={"ID":"1d89df17-68ce-46c6-8d9c-3de3b489acb1","Type":"ContainerDied","Data":"d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c"}
Jul 27 16:52:11.297801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.297706    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"
Jul 27 16:52:11.297801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.297731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf" event={"ID":"1d89df17-68ce-46c6-8d9c-3de3b489acb1","Type":"ContainerDied","Data":"786d9a7a7b8c3054b80cb3a5186cec4a25d6fa1cca41f89c198fe4770f4baaed"}
Jul 27 16:52:11.305241 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.305224    2559 scope.go:117] "RemoveContainer" containerID="9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0"
Jul 27 16:52:11.305561 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:52:11.305539    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0\": container with ID starting with 9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0 not found: ID does not exist" containerID="9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0"
Jul 27 16:52:11.305642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.305567    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0"} err="failed to get container status \"9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0\": rpc error: code = NotFound desc = could not find container \"9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0\": container with ID starting with 9da3c3f8281d8d3ca239f5792e4ec2179703756d744dbaff6547aab6b6910db0 not found: ID does not exist"
Jul 27 16:52:11.305642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.305583    2559 scope.go:117] "RemoveContainer" containerID="df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b"
Jul 27 16:52:11.313482 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.313466    2559 scope.go:117] "RemoveContainer" containerID="df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b"
Jul 27 16:52:11.313715 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:52:11.313696    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b\": container with ID starting with df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b not found: ID does not exist" containerID="df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b"
Jul 27 16:52:11.313783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.313722    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b"} err="failed to get container status \"df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b\": rpc error: code = NotFound desc = could not find container \"df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b\": container with ID starting with df87466e1929b1721306bcdefda1502f1cd0e4bf43dc39d96ef1ca17c2637e5b not found: ID does not exist"
Jul 27 16:52:11.313783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.313745    2559 scope.go:117] "RemoveContainer" containerID="d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c"
Jul 27 16:52:11.320718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.320692    2559 scope.go:117] "RemoveContainer" containerID="d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c"
Jul 27 16:52:11.320964 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:52:11.320947    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c\": container with ID starting with d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c not found: ID does not exist" containerID="d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c"
Jul 27 16:52:11.321004 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.320971    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c"} err="failed to get container status \"d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c\": rpc error: code = NotFound desc = could not find container \"d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c\": container with ID starting with d436bbc6ea2ac6649e81cfefd23233fb6546bdb49cc7750e13306a6e8cc6267c not found: ID does not exist"
Jul 27 16:52:11.323702 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.323681    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"]
Jul 27 16:52:11.326939 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.326917    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fee06a22-5697d8d9bd-rr4jt"]
Jul 27 16:52:11.339431 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.339411    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"]
Jul 27 16:52:11.346170 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.346150    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fee06a22-55689b7fcd-fzj2v"]
Jul 27 16:52:11.360093 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.360066    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"]
Jul 27 16:52:11.368707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.368684    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fee06a22-7fb7b4cf7b-7r6vf"]
Jul 27 16:52:11.381649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.381628    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsjc\" (UniqueName: \"kubernetes.io/projected/1d89df17-68ce-46c6-8d9c-3de3b489acb1-kube-api-access-snsjc\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:11.381722 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:11.381649    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d89df17-68ce-46c6-8d9c-3de3b489acb1-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:12.735922 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:12.735885    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d89df17-68ce-46c6-8d9c-3de3b489acb1" path="/var/lib/kubelet/pods/1d89df17-68ce-46c6-8d9c-3de3b489acb1/volumes"
Jul 27 16:52:12.736332 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:12.736315    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864aeb5f-09d8-479d-9f63-23fc78bbbf81" path="/var/lib/kubelet/pods/864aeb5f-09d8-479d-9f63-23fc78bbbf81/volumes"
Jul 27 16:52:12.736671 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:12.736658    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872aa7f7-e836-40b6-8db1-2b87916fa470" path="/var/lib/kubelet/pods/872aa7f7-e836-40b6-8db1-2b87916fa470/volumes"
Jul 27 16:52:15.514099 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:15.514054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:15.946655 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:15.946621    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:16.462935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:16.462869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:16.569377 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:16.569341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:21.981614 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:21.981578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:22.147362 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.147326    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:22.280606 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.280512    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"]
Jul 27 16:52:22.280834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.280807    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy" containerID="cri-o://4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64" gracePeriod=30
Jul 27 16:52:22.288616 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.288591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:22.528893 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.528860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:22.689002 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.688969    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:22.773390 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:22.773355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:23.050159 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:23.050078    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:27.529298 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.529249    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:52:27.627399 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627305    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-envoy\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627399 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-istio-token\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627399 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627388    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t4lk\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-kube-api-access-5t4lk\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627689 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-certs\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627689 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be63c8e4-d1e5-46f0-b307-437646b71986-istio-podinfo\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627689 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-credential-socket\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627689 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-socket\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627689 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627563    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-data\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.627914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627792    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:52:27.627914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627835    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:52:27.627914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627855    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:52:27.627914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.627912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be63c8e4-d1e5-46f0-b307-437646b71986-istiod-ca-cert\") pod \"be63c8e4-d1e5-46f0-b307-437646b71986\" (UID: \"be63c8e4-d1e5-46f0-b307-437646b71986\") "
Jul 27 16:52:27.628234 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.628216    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.628320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.628240    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.628320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.628254    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.628423 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.628336    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be63c8e4-d1e5-46f0-b307-437646b71986-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:52:27.628691 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.628671    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-data" (OuterVolumeSpecName: "istio-data") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:52:27.629617 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.629594    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-istio-token" (OuterVolumeSpecName: "istio-token") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:52:27.629701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.629647    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-kube-api-access-5t4lk" (OuterVolumeSpecName: "kube-api-access-5t4lk") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "kube-api-access-5t4lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:52:27.629935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.629913    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:52:27.630033 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.629925    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be63c8e4-d1e5-46f0-b307-437646b71986-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be63c8e4-d1e5-46f0-b307-437646b71986" (UID: "be63c8e4-d1e5-46f0-b307-437646b71986"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:52:27.729003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.728965    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.729003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.728997    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be63c8e4-d1e5-46f0-b307-437646b71986-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.729003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.729006    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be63c8e4-d1e5-46f0-b307-437646b71986-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.729003 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.729014    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.729262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.729023    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t4lk\" (UniqueName: \"kubernetes.io/projected/be63c8e4-d1e5-46f0-b307-437646b71986-kube-api-access-5t4lk\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:27.729262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:27.729032    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be63c8e4-d1e5-46f0-b307-437646b71986-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:52:28.374994 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.374961    2559 generic.go:356] "Generic (PLEG): container finished" podID="be63c8e4-d1e5-46f0-b307-437646b71986" containerID="4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64" exitCode=0
Jul 27 16:52:28.375164 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.375008    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"
Jul 27 16:52:28.375164 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.375047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" event={"ID":"be63c8e4-d1e5-46f0-b307-437646b71986","Type":"ContainerDied","Data":"4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64"}
Jul 27 16:52:28.375164 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.375102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z" event={"ID":"be63c8e4-d1e5-46f0-b307-437646b71986","Type":"ContainerDied","Data":"35a9ba0815d22f545d7e767677254b1419bcd5371233986211006f93906d74e8"}
Jul 27 16:52:28.375164 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.375125    2559 scope.go:117] "RemoveContainer" containerID="4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64"
Jul 27 16:52:28.383572 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.383554    2559 scope.go:117] "RemoveContainer" containerID="1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"
Jul 27 16:52:28.390941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.390924    2559 scope.go:117] "RemoveContainer" containerID="4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64"
Jul 27 16:52:28.391177 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:52:28.391160    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64\": container with ID starting with 4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64 not found: ID does not exist" containerID="4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64"
Jul 27 16:52:28.391238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.391184    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64"} err="failed to get container status \"4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64\": rpc error: code = NotFound desc = could not find container \"4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64\": container with ID starting with 4b39ae7113ea288e020b4fce1a976cdcc7d4f2ddd4248f4bc45c6911844d1b64 not found: ID does not exist"
Jul 27 16:52:28.391238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.391201    2559 scope.go:117] "RemoveContainer" containerID="1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"
Jul 27 16:52:28.391459 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:52:28.391444    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f\": container with ID starting with 1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f not found: ID does not exist" containerID="1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"
Jul 27 16:52:28.391504 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.391466    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f"} err="failed to get container status \"1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f\": rpc error: code = NotFound desc = could not find container \"1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f\": container with ID starting with 1e3c86e8baa841c7591900747aa1b56edbc37d2aca64e128a6e4018c10ee3a4f not found: ID does not exist"
Jul 27 16:52:28.402557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.402534    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"]
Jul 27 16:52:28.407437 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.407411    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fee06a22-openshift-default-7f8b8b485b-fh98z"]
Jul 27 16:52:28.735415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.735381    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" path="/var/lib/kubelet/pods/be63c8e4-d1e5-46f0-b307-437646b71986/volumes"
Jul 27 16:52:28.799473 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.799439    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:28.869016 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.868979    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:28.955450 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:28.955422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:29.056059 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:29.055977    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:29.164653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:29.164624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:35.354087 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:35.354009    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:35.440069 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:35.440034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:42.296768 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.296733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:42.585956 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.585874    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"]
Jul 27 16:52:42.586190 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586178    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d89df17-68ce-46c6-8d9c-3de3b489acb1" containerName="payload-pre-processing"
Jul 27 16:52:42.586231 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d89df17-68ce-46c6-8d9c-3de3b489acb1" containerName="payload-pre-processing"
Jul 27 16:52:42.586231 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586201    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy"
Jul 27 16:52:42.586231 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586206    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy"
Jul 27 16:52:42.586231 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872aa7f7-e836-40b6-8db1-2b87916fa470" containerName="payload-processing"
Jul 27 16:52:42.586231 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586222    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="872aa7f7-e836-40b6-8db1-2b87916fa470" containerName="payload-processing"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586239    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864aeb5f-09d8-479d-9f63-23fc78bbbf81" containerName="maas-api"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586244    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="864aeb5f-09d8-479d-9f63-23fc78bbbf81" containerName="maas-api"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586325    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586332    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="864aeb5f-09d8-479d-9f63-23fc78bbbf81" containerName="maas-api"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586339    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="872aa7f7-e836-40b6-8db1-2b87916fa470" containerName="payload-processing"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586347    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy"
Jul 27 16:52:42.586410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.586353    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d89df17-68ce-46c6-8d9c-3de3b489acb1" containerName="payload-pre-processing"
Jul 27 16:52:42.590598 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.590581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.597338 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.597310    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dc13869f-openshift-default-dockercfg-l5xmb\""
Jul 27 16:52:42.607517 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.607491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"]
Jul 27 16:52:42.637898 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.637864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:42.761736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761694    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-certs\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.761921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-credential-socket\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.761921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761775    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-envoy\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.761921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761829    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-istio-token\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.761921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-data\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.761921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ca297-b721-46f6-b71a-43dd7beda99b-istiod-ca-cert\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.761921 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-socket\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.762185 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.761974    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ca297-b721-46f6-b71a-43dd7beda99b-istio-podinfo\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.762185 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.762009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86l4\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-kube-api-access-b86l4\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ca297-b721-46f6-b71a-43dd7beda99b-istio-podinfo\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86l4\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-kube-api-access-b86l4\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-certs\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-credential-socket\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-envoy\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863695 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-istio-token\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863695 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-data\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863695 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ca297-b721-46f6-b71a-43dd7beda99b-istiod-ca-cert\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863849 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-socket\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863849 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-certs\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.863954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-data\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.864014 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.863952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-credential-socket\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.864070 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.864056    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-socket\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.864166 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.864145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ca297-b721-46f6-b71a-43dd7beda99b-istiod-ca-cert\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.865659 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.865636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-envoy\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.865775 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.865748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ca297-b721-46f6-b71a-43dd7beda99b-istio-podinfo\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.872128 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.872105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-istio-token\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.872436 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.872378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86l4\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-kube-api-access-b86l4\") pod \"e2e-mt-dc13869f-openshift-default-575748bb48-kld5t\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") " pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.903358 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.903320    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:42.986782 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:42.986727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:43.038575 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.038501    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"]
Jul 27 16:52:43.041648 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:52:43.041605    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780ca297_b721_46f6_b71a_43dd7beda99b.slice/crio-eaa6166a3853b2a2f1cc01711887764fd8e4de52d680e0b273ec407409d61c66 WatchSource:0}: Error finding container eaa6166a3853b2a2f1cc01711887764fd8e4de52d680e0b273ec407409d61c66: Status 404 returned error can't find the container with id eaa6166a3853b2a2f1cc01711887764fd8e4de52d680e0b273ec407409d61c66
Jul 27 16:52:43.043511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.043480    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:43.043579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.043548    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:43.043579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.043576    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:43.440686 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.440646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" event={"ID":"780ca297-b721-46f6-b71a-43dd7beda99b","Type":"ContainerStarted","Data":"6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"}
Jul 27 16:52:43.441071 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.440700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" event={"ID":"780ca297-b721-46f6-b71a-43dd7beda99b","Type":"ContainerStarted","Data":"eaa6166a3853b2a2f1cc01711887764fd8e4de52d680e0b273ec407409d61c66"}
Jul 27 16:52:43.465817 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.465757    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" podStartSLOduration=1.465733432 podStartE2EDuration="1.465733432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:52:43.463922063 +0000 UTC m=+2163.365062510" watchObservedRunningTime="2026-07-27 16:52:43.465733432 +0000 UTC m=+2163.366873877"
Jul 27 16:52:43.903675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:43.903642    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:44.909070 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:44.909041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:45.382709 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.382607    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:45.450229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.450197    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:45.451156 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.451137    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:45.518763 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.518725    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"]
Jul 27 16:52:45.519206 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.519190    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy"
Jul 27 16:52:45.519260 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.519211    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="be63c8e4-d1e5-46f0-b307-437646b71986" containerName="istio-proxy"
Jul 27 16:52:45.522432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.522412    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.528805 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.528781    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dc13869f-serving-cert\""
Jul 27 16:52:45.529044 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.529022    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:52:45.529336 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.529314    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:52:45.529845 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.529825    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:52:45.545319 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.545270    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"]
Jul 27 16:52:45.572359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.572322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:45.646595 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.646549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:45.688824 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.688781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8zr\" (UniqueName: \"kubernetes.io/projected/94996d7a-f05b-4ab0-9127-766d2c338168-kube-api-access-6k8zr\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.688996 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.688836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94996d7a-f05b-4ab0-9127-766d2c338168-maas-api-tls\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.688996 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.688923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94996d7a-f05b-4ab0-9127-766d2c338168-openshift-service-ca\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.789914 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.789875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8zr\" (UniqueName: \"kubernetes.io/projected/94996d7a-f05b-4ab0-9127-766d2c338168-kube-api-access-6k8zr\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.790099 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.789923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94996d7a-f05b-4ab0-9127-766d2c338168-maas-api-tls\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.790099 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.789956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94996d7a-f05b-4ab0-9127-766d2c338168-openshift-service-ca\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.790579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.790555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94996d7a-f05b-4ab0-9127-766d2c338168-openshift-service-ca\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.792413 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.792393    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94996d7a-f05b-4ab0-9127-766d2c338168-maas-api-tls\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.799134 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.799113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8zr\" (UniqueName: \"kubernetes.io/projected/94996d7a-f05b-4ab0-9127-766d2c338168-kube-api-access-6k8zr\") pod \"maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.802453 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.802431    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:45.835478 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.835435    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:45.974449 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:45.974427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"]
Jul 27 16:52:45.977206 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:52:45.977178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94996d7a_f05b_4ab0_9127_766d2c338168.slice/crio-a797ffe5dd2c1b9fdb903d68bed2694bff40649e6bc8e322bf686e667e00c19d WatchSource:0}: Error finding container a797ffe5dd2c1b9fdb903d68bed2694bff40649e6bc8e322bf686e667e00c19d: Status 404 returned error can't find the container with id a797ffe5dd2c1b9fdb903d68bed2694bff40649e6bc8e322bf686e667e00c19d
Jul 27 16:52:46.079963 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.079919    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:46.177460 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.177376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:46.194451 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.194412    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"]
Jul 27 16:52:46.198632 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.198609    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.201402 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.201374    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dc13869f\""
Jul 27 16:52:46.201522 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.201447    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dc13869f-dockercfg-vcnjk\""
Jul 27 16:52:46.209913 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.209886    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"]
Jul 27 16:52:46.294509 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.294472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.294509 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.294527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwlt\" (UniqueName: \"kubernetes.io/projected/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-kube-api-access-fwwlt\") pod \"payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.395874 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.395826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwlt\" (UniqueName: \"kubernetes.io/projected/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-kube-api-access-fwwlt\") pod \"payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.396085 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.395923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.396684 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.396658    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.406622 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.406600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwlt\" (UniqueName: \"kubernetes.io/projected/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-kube-api-access-fwwlt\") pod \"payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.455516 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.455430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9" event={"ID":"94996d7a-f05b-4ab0-9127-766d2c338168","Type":"ContainerStarted","Data":"a797ffe5dd2c1b9fdb903d68bed2694bff40649e6bc8e322bf686e667e00c19d"}
Jul 27 16:52:46.508956 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.508912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:46.641419 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:46.641391    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"]
Jul 27 16:52:46.643863 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:52:46.643801    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb8cc4a_d362_4856_8e86_64ecac9a9bb0.slice/crio-feeb9041f8987dc9ab45fb16084953ed5008c809a90334ded8674555ddac6d7a WatchSource:0}: Error finding container feeb9041f8987dc9ab45fb16084953ed5008c809a90334ded8674555ddac6d7a: Status 404 returned error can't find the container with id feeb9041f8987dc9ab45fb16084953ed5008c809a90334ded8674555ddac6d7a
Jul 27 16:52:47.460748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:47.460708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7" event={"ID":"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0","Type":"ContainerStarted","Data":"2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c"}
Jul 27 16:52:47.461183 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:47.460776    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:47.461183 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:47.460800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7" event={"ID":"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0","Type":"ContainerStarted","Data":"feeb9041f8987dc9ab45fb16084953ed5008c809a90334ded8674555ddac6d7a"}
Jul 27 16:52:47.462320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:47.462294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9" event={"ID":"94996d7a-f05b-4ab0-9127-766d2c338168","Type":"ContainerStarted","Data":"9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986"}
Jul 27 16:52:47.480664 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:47.480614    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7" podStartSLOduration=1.480598741 podStartE2EDuration="1.480598741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:52:47.478588079 +0000 UTC m=+2167.379728538" watchObservedRunningTime="2026-07-27 16:52:47.480598741 +0000 UTC m=+2167.381739242"
Jul 27 16:52:47.500875 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:47.500818    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9" podStartSLOduration=1.89013572 podStartE2EDuration="2.500799027s" totalImagesPullingTime="610.663307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:52:47.499671624 +0000 UTC m=+2167.400812071" watchObservedRunningTime="2026-07-27 16:52:47.500799027 +0000 UTC m=+2167.401939473"
Jul 27 16:52:48.466955 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:48.466921    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:50.030135 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:50.030097    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:54.476956 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:54.476922    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:52:57.389210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:57.389178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:52:58.469342 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.469276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:52:58.511992 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.511924    2559 generic.go:356] "Generic (PLEG): container finished" podID="780ca297-b721-46f6-b71a-43dd7beda99b" containerID="6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101" exitCode=0
Jul 27 16:52:58.512129 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.511997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" event={"ID":"780ca297-b721-46f6-b71a-43dd7beda99b","Type":"ContainerDied","Data":"6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"}
Jul 27 16:52:58.512549 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.512533    2559 scope.go:117] "RemoveContainer" containerID="6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"
Jul 27 16:52:58.513521 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.513496    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:58.513618 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.513575    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:58.513677 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:58.513626    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:52:59.518108 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:59.518070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" event={"ID":"780ca297-b721-46f6-b71a-43dd7beda99b","Type":"ContainerStarted","Data":"21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2"}
Jul 27 16:52:59.904045 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:59.904007    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:52:59.904755 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:59.904728    2559 patch_prober.go:28] interesting pod/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 27 16:52:59.904882 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:52:59.904780    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 27 16:53:00.450788 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:00.450730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:53:00.903931 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:00.903892    2559 patch_prober.go:28] interesting pod/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 27 16:53:00.904444 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:00.903977    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 27 16:53:01.904792 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:01.904742    2559 patch_prober.go:28] interesting pod/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 27 16:53:01.905196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:01.904809    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 27 16:53:02.904019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:02.903980    2559 patch_prober.go:28] interesting pod/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 27 16:53:02.904200 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:02.904040    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 27 16:53:03.908365 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:03.908331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:53:03.909246 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:03.909225    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:53:41.532802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:41.532770    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:53:45.164142 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.163258    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"]
Jul 27 16:53:45.164142 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.163511    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9" podUID="94996d7a-f05b-4ab0-9127-766d2c338168" containerName="maas-api" containerID="cri-o://9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986" gracePeriod=30
Jul 27 16:53:45.195757 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.195729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:53:45.237628 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.237591    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"]
Jul 27 16:53:45.237866 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.237841    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7" podUID="4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" containerName="payload-pre-processing" containerID="cri-o://2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c" gracePeriod=30
Jul 27 16:53:45.264196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.264162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:53:45.322735 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.322675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:53:45.517901 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.517812    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:53:45.530488 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.530466    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:53:45.607508 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.607475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94996d7a-f05b-4ab0-9127-766d2c338168-maas-api-tls\") pod \"94996d7a-f05b-4ab0-9127-766d2c338168\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") "
Jul 27 16:53:45.607508 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.607509    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94996d7a-f05b-4ab0-9127-766d2c338168-openshift-service-ca\") pod \"94996d7a-f05b-4ab0-9127-766d2c338168\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") "
Jul 27 16:53:45.607714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.607552    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwwlt\" (UniqueName: \"kubernetes.io/projected/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-kube-api-access-fwwlt\") pod \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") "
Jul 27 16:53:45.607714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.607668    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-plugins-config-volume\") pod \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\" (UID: \"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0\") "
Jul 27 16:53:45.607823 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.607725    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8zr\" (UniqueName: \"kubernetes.io/projected/94996d7a-f05b-4ab0-9127-766d2c338168-kube-api-access-6k8zr\") pod \"94996d7a-f05b-4ab0-9127-766d2c338168\" (UID: \"94996d7a-f05b-4ab0-9127-766d2c338168\") "
Jul 27 16:53:45.608016 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.607983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94996d7a-f05b-4ab0-9127-766d2c338168-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94996d7a-f05b-4ab0-9127-766d2c338168" (UID: "94996d7a-f05b-4ab0-9127-766d2c338168"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:53:45.608138 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.608026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" (UID: "4cb8cc4a-d362-4856-8e86-64ecac9a9bb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:53:45.609636 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.609609    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94996d7a-f05b-4ab0-9127-766d2c338168-kube-api-access-6k8zr" (OuterVolumeSpecName: "kube-api-access-6k8zr") pod "94996d7a-f05b-4ab0-9127-766d2c338168" (UID: "94996d7a-f05b-4ab0-9127-766d2c338168"). InnerVolumeSpecName "kube-api-access-6k8zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:53:45.609636 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.609623    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94996d7a-f05b-4ab0-9127-766d2c338168-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94996d7a-f05b-4ab0-9127-766d2c338168" (UID: "94996d7a-f05b-4ab0-9127-766d2c338168"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:53:45.609777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.609653    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-kube-api-access-fwwlt" (OuterVolumeSpecName: "kube-api-access-fwwlt") pod "4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" (UID: "4cb8cc4a-d362-4856-8e86-64ecac9a9bb0"). InnerVolumeSpecName "kube-api-access-fwwlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:53:45.708357 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.708251    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94996d7a-f05b-4ab0-9127-766d2c338168-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:53:45.708357 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.708296    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94996d7a-f05b-4ab0-9127-766d2c338168-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:53:45.708357 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.708307    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwlt\" (UniqueName: \"kubernetes.io/projected/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-kube-api-access-fwwlt\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:53:45.708357 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.708315    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:53:45.708357 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.708324    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8zr\" (UniqueName: \"kubernetes.io/projected/94996d7a-f05b-4ab0-9127-766d2c338168-kube-api-access-6k8zr\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:53:45.720506 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.720472    2559 generic.go:356] "Generic (PLEG): container finished" podID="94996d7a-f05b-4ab0-9127-766d2c338168" containerID="9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986" exitCode=0
Jul 27 16:53:45.720649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.720515    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"
Jul 27 16:53:45.720649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.720567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9" event={"ID":"94996d7a-f05b-4ab0-9127-766d2c338168","Type":"ContainerDied","Data":"9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986"}
Jul 27 16:53:45.720649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.720612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9" event={"ID":"94996d7a-f05b-4ab0-9127-766d2c338168","Type":"ContainerDied","Data":"a797ffe5dd2c1b9fdb903d68bed2694bff40649e6bc8e322bf686e667e00c19d"}
Jul 27 16:53:45.720649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.720632    2559 scope.go:117] "RemoveContainer" containerID="9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986"
Jul 27 16:53:45.728425 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.728396    2559 generic.go:356] "Generic (PLEG): container finished" podID="4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" containerID="2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c" exitCode=0
Jul 27 16:53:45.728528 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.728460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7" event={"ID":"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0","Type":"ContainerDied","Data":"2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c"}
Jul 27 16:53:45.728575 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.728519    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"
Jul 27 16:53:45.728575 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.728530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7" event={"ID":"4cb8cc4a-d362-4856-8e86-64ecac9a9bb0","Type":"ContainerDied","Data":"feeb9041f8987dc9ab45fb16084953ed5008c809a90334ded8674555ddac6d7a"}
Jul 27 16:53:45.736111 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.736095    2559 scope.go:117] "RemoveContainer" containerID="9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986"
Jul 27 16:53:45.736442 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:53:45.736420    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986\": container with ID starting with 9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986 not found: ID does not exist" containerID="9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986"
Jul 27 16:53:45.736559 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.736451    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986"} err="failed to get container status \"9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986\": rpc error: code = NotFound desc = could not find container \"9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986\": container with ID starting with 9c007721968a6f9c1cde376ec508e677755f190c7ed086166f9b9785a61ea986 not found: ID does not exist"
Jul 27 16:53:45.736559 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.736476    2559 scope.go:117] "RemoveContainer" containerID="2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c"
Jul 27 16:53:45.744324 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.744301    2559 scope.go:117] "RemoveContainer" containerID="2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c"
Jul 27 16:53:45.744771 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:53:45.744592    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c\": container with ID starting with 2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c not found: ID does not exist" containerID="2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c"
Jul 27 16:53:45.744771 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.744631    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c"} err="failed to get container status \"2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c\": rpc error: code = NotFound desc = could not find container \"2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c\": container with ID starting with 2fd76b8e01c66c6839f1c0b703c20421501b2268e298f5e01702c7c9ce3fe41c not found: ID does not exist"
Jul 27 16:53:45.746401 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.746383    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"]
Jul 27 16:53:45.750498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.750478    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc13869f-685f794dd9-x9bd9"]
Jul 27 16:53:45.763578 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.763555    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"]
Jul 27 16:53:45.766971 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:45.766950    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dc13869f-58fbb4bb4-qjdd7"]
Jul 27 16:53:46.734707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:46.734673    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" path="/var/lib/kubelet/pods/4cb8cc4a-d362-4856-8e86-64ecac9a9bb0/volumes"
Jul 27 16:53:46.735239 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:46.735218    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94996d7a-f05b-4ab0-9127-766d2c338168" path="/var/lib/kubelet/pods/94996d7a-f05b-4ab0-9127-766d2c338168/volumes"
Jul 27 16:53:49.674452 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:49.674420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:53:56.389063 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:56.389023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:53:58.007448 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:53:58.007417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:35.175145 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:35.175108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:36.010966 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:36.010931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:42.613400 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:42.613349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:42.692091 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:42.692044    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:43.005069 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:43.005029    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"]
Jul 27 16:54:43.005303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:43.005246    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy" containerID="cri-o://21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2" gracePeriod=30
Jul 27 16:54:43.027325 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:43.027275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:43.223860 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:43.223831    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:44.447834 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:44.447795    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:44.538189 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:44.538159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:45.393203 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:45.393161    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:45.495659 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:45.495628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:46.545133 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:46.545098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:47.195652 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:47.195618    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:48.252745 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.252715    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:54:48.412825 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.412790    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ca297-b721-46f6-b71a-43dd7beda99b-istio-podinfo\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.412882    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-credential-socket\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.412923    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-certs\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.412944    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-socket\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.412980    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b86l4\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-kube-api-access-b86l4\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ca297-b721-46f6-b71a-43dd7beda99b-istiod-ca-cert\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413068    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-data\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413096    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-envoy\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413144    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-istio-token\") pod \"780ca297-b721-46f6-b71a-43dd7beda99b\" (UID: \"780ca297-b721-46f6-b71a-43dd7beda99b\") "
Jul 27 16:54:48.413511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:48.413511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413229    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:48.413511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413436    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.413511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413457    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.413717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413533    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780ca297-b721-46f6-b71a-43dd7beda99b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:54:48.413717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.413273    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:48.414253 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.414219    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-data" (OuterVolumeSpecName: "istio-data") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:48.415025 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.414998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/780ca297-b721-46f6-b71a-43dd7beda99b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:54:48.415324 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.415275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-kube-api-access-b86l4" (OuterVolumeSpecName: "kube-api-access-b86l4") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "kube-api-access-b86l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:54:48.415410 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.415360    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:48.415498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.415481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-istio-token" (OuterVolumeSpecName: "istio-token") pod "780ca297-b721-46f6-b71a-43dd7beda99b" (UID: "780ca297-b721-46f6-b71a-43dd7beda99b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:54:48.514432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514390    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/780ca297-b721-46f6-b71a-43dd7beda99b-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.514432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514423    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.514432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514435    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.514681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514445    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.514681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514456    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/780ca297-b721-46f6-b71a-43dd7beda99b-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.514681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514467    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/780ca297-b721-46f6-b71a-43dd7beda99b-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.514681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.514481    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86l4\" (UniqueName: \"kubernetes.io/projected/780ca297-b721-46f6-b71a-43dd7beda99b-kube-api-access-b86l4\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:54:48.992176 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.992137    2559 generic.go:356] "Generic (PLEG): container finished" podID="780ca297-b721-46f6-b71a-43dd7beda99b" containerID="21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2" exitCode=0
Jul 27 16:54:48.992412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.992210    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"
Jul 27 16:54:48.992412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.992221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" event={"ID":"780ca297-b721-46f6-b71a-43dd7beda99b","Type":"ContainerDied","Data":"21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2"}
Jul 27 16:54:48.992412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.992311    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t" event={"ID":"780ca297-b721-46f6-b71a-43dd7beda99b","Type":"ContainerDied","Data":"eaa6166a3853b2a2f1cc01711887764fd8e4de52d680e0b273ec407409d61c66"}
Jul 27 16:54:48.992412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:48.992335    2559 scope.go:117] "RemoveContainer" containerID="21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2"
Jul 27 16:54:49.000738 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.000698    2559 scope.go:117] "RemoveContainer" containerID="6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"
Jul 27 16:54:49.008171 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.008152    2559 scope.go:117] "RemoveContainer" containerID="21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2"
Jul 27 16:54:49.008439 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:54:49.008419    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2\": container with ID starting with 21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2 not found: ID does not exist" containerID="21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2"
Jul 27 16:54:49.008490 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.008448    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2"} err="failed to get container status \"21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2\": rpc error: code = NotFound desc = could not find container \"21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2\": container with ID starting with 21e053ecf03968ebd7fe00d401bbb6675361128f327984ed21e282b7800e57e2 not found: ID does not exist"
Jul 27 16:54:49.008490 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.008466    2559 scope.go:117] "RemoveContainer" containerID="6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"
Jul 27 16:54:49.008709 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:54:49.008695    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101\": container with ID starting with 6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101 not found: ID does not exist" containerID="6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"
Jul 27 16:54:49.008791 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.008713    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101"} err="failed to get container status \"6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101\": rpc error: code = NotFound desc = could not find container \"6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101\": container with ID starting with 6e97d0a514cb9e6afd0ff7d8356a83bbaae16bd8a5b2be7d6974f743df81f101 not found: ID does not exist"
Jul 27 16:54:49.013009 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.012985    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"]
Jul 27 16:54:49.017774 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:49.017743    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dc13869f-openshift-default-575748bb48-kld5t"]
Jul 27 16:54:50.735560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:50.735524    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" path="/var/lib/kubelet/pods/780ca297-b721-46f6-b71a-43dd7beda99b/volumes"
Jul 27 16:54:54.012202 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:54.012159    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:54.121802 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:54.121768    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:54.472224 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:54.472189    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:54:58.298496 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:54:58.298459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:01.118322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:01.118273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:01.267560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:01.267526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:01.349763 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:01.349729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:11.885945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:11.885912    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:12.240821 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:12.240784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:22.849935 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:22.849899    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:33.228328 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:33.228231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:40.159850 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.159813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:40.252229 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.252195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:40.519979 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.519862    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:40.779980 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.779907    2559 scope.go:117] "RemoveContainer" containerID="9051d3074175d72798a9524c1ff2d23f6ff3dd623c1dbadd14d43b19fb2af574"
Jul 27 16:55:40.841994 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.841964    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"]
Jul 27 16:55:40.842271 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842259    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" containerName="payload-pre-processing"
Jul 27 16:55:40.842327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842272    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" containerName="payload-pre-processing"
Jul 27 16:55:40.842327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842298    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy"
Jul 27 16:55:40.842327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842304    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy"
Jul 27 16:55:40.842327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842314    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94996d7a-f05b-4ab0-9127-766d2c338168" containerName="maas-api"
Jul 27 16:55:40.842327 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842319    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="94996d7a-f05b-4ab0-9127-766d2c338168" containerName="maas-api"
Jul 27 16:55:40.842475 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842367    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="94996d7a-f05b-4ab0-9127-766d2c338168" containerName="maas-api"
Jul 27 16:55:40.842475 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842377    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb8cc4a-d362-4856-8e86-64ecac9a9bb0" containerName="payload-pre-processing"
Jul 27 16:55:40.842475 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842382    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy"
Jul 27 16:55:40.842475 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.842388    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy"
Jul 27 16:55:40.846232 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.846214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.849383 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.849360    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-23e152-openshift-default-dockercfg-42zsk\""
Jul 27 16:55:40.857826 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.857803    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"]
Jul 27 16:55:40.864371 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.864347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:40.924967 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.924933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-data\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-certs\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-token\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istiod-ca-cert\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-credential-socket\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-envoy\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g64s6\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-kube-api-access-g64s6\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-socket\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:40.925425 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:40.925353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-podinfo\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026128 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-podinfo\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026128 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-data\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-certs\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-token\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istiod-ca-cert\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-credential-socket\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-envoy\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g64s6\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-kube-api-access-g64s6\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026415 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-socket\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026781 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-data\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026781 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-credential-socket\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.026781 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.026763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-socket\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.027039 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.027016    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istiod-ca-cert\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.027102 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.027053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-certs\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.028448 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.028432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-envoy\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.028751 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.028732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-podinfo\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.035542 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.035473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-token\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.035665 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.035647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g64s6\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-kube-api-access-g64s6\") pod \"e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") " pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.155833 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.155803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:41.312595 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.312418    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"]
Jul 27 16:55:41.314951 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:55:41.314920    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08faa6a3_a0a9_4baf_a9d0_d4638a771324.slice/crio-055483c5433ccf0b75acc08aad10180bb8f10b1d4b543dfb69f93da538cf26a9 WatchSource:0}: Error finding container 055483c5433ccf0b75acc08aad10180bb8f10b1d4b543dfb69f93da538cf26a9: Status 404 returned error can't find the container with id 055483c5433ccf0b75acc08aad10180bb8f10b1d4b543dfb69f93da538cf26a9
Jul 27 16:55:41.316989 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.316956    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:55:41.317059 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.317046    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:55:41.317098 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.317075    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:55:41.484329 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:41.484272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:42.065972 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:42.065926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:42.176972 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:42.176929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:42.217470 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:42.217438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" event={"ID":"08faa6a3-a0a9-4baf-a9d0-d4638a771324","Type":"ContainerStarted","Data":"9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"}
Jul 27 16:55:42.217635 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:42.217485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" event={"ID":"08faa6a3-a0a9-4baf-a9d0-d4638a771324","Type":"ContainerStarted","Data":"055483c5433ccf0b75acc08aad10180bb8f10b1d4b543dfb69f93da538cf26a9"}
Jul 27 16:55:42.252511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:42.252464    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podStartSLOduration=2.25244956 podStartE2EDuration="2.25244956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:42.249702328 +0000 UTC m=+2342.150842774" watchObservedRunningTime="2026-07-27 16:55:42.25244956 +0000 UTC m=+2342.153590006"
Jul 27 16:55:43.156602 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:43.156571    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:43.161460 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:43.161434    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:43.221702 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:43.221671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:43.222493 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:43.222473    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:44.288632 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.288576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:44.391976 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.391942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:44.401730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.401702    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"]
Jul 27 16:55:44.402095 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.402076    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy"
Jul 27 16:55:44.402171 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.402098    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="780ca297-b721-46f6-b71a-43dd7beda99b" containerName="istio-proxy"
Jul 27 16:55:44.405208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.405187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.408243 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.408223    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:55:44.409422 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.409404    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:55:44.409531 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.409406    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-23e152-serving-cert\""
Jul 27 16:55:44.409531 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.409446    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:55:44.419079 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.419061    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"]
Jul 27 16:55:44.487597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.487563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:44.556816 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.556727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.556972 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.556841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-maas-api-tls\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.556972 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.556869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvp6\" (UniqueName: \"kubernetes.io/projected/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-kube-api-access-4dvp6\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.600511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.600481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:44.657670 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.657640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.657821 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.657726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-maas-api-tls\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.657821 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.657747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvp6\" (UniqueName: \"kubernetes.io/projected/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-kube-api-access-4dvp6\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.658245 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.658223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.660022 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.660001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-maas-api-tls\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.671994 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.671963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvp6\" (UniqueName: \"kubernetes.io/projected/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-kube-api-access-4dvp6\") pod \"maas-api-e2e-inf-a-23e152-675f696dd5-xbffg\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.715855 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.715830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:44.855888 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:44.855859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"]
Jul 27 16:55:44.858174 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:55:44.858149    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26b937a_1d7c_4e07_b6d7_b0ef83f43bfd.slice/crio-f2ba9b7d42d90eaf927c39e2c9af6c8f3bf0d5b4e86803a7b509fec9a064db3d WatchSource:0}: Error finding container f2ba9b7d42d90eaf927c39e2c9af6c8f3bf0d5b4e86803a7b509fec9a064db3d: Status 404 returned error can't find the container with id f2ba9b7d42d90eaf927c39e2c9af6c8f3bf0d5b4e86803a7b509fec9a064db3d
Jul 27 16:55:45.059667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.059629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:45.160560 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.160529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:45.196759 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.196726    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"]
Jul 27 16:55:45.201335 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.201317    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.204849 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.204823    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-23e152\""
Jul 27 16:55:45.205757 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.205738    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-23e152-dockercfg-nwwfw\""
Jul 27 16:55:45.217072 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.217052    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"]
Jul 27 16:55:45.232886 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.232852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg" event={"ID":"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd","Type":"ContainerStarted","Data":"f2ba9b7d42d90eaf927c39e2c9af6c8f3bf0d5b4e86803a7b509fec9a064db3d"}
Jul 27 16:55:45.262603 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.262577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-23e152-759f974b-nv48h\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.262708 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.262624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2hwq\" (UniqueName: \"kubernetes.io/projected/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-kube-api-access-c2hwq\") pod \"payload-processing-e2e-inf-a-23e152-759f974b-nv48h\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.267021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.266999    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"]
Jul 27 16:55:45.270165 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.270151    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.290677 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.290654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"]
Jul 27 16:55:45.363245 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.363215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-23e152-759f974b-nv48h\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.363420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.363324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2hwq\" (UniqueName: \"kubernetes.io/projected/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-kube-api-access-c2hwq\") pod \"payload-processing-e2e-inf-a-23e152-759f974b-nv48h\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.363420 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.363364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac706e0-2b30-4418-a25d-b5635f50f269-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.363511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.363416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4h4s\" (UniqueName: \"kubernetes.io/projected/7ac706e0-2b30-4418-a25d-b5635f50f269-kube-api-access-q4h4s\") pod \"payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.364334 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.364312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-23e152-759f974b-nv48h\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.378093 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.378065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2hwq\" (UniqueName: \"kubernetes.io/projected/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-kube-api-access-c2hwq\") pod \"payload-processing-e2e-inf-a-23e152-759f974b-nv48h\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.464777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.464691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac706e0-2b30-4418-a25d-b5635f50f269-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.464777 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.464745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4h4s\" (UniqueName: \"kubernetes.io/projected/7ac706e0-2b30-4418-a25d-b5635f50f269-kube-api-access-q4h4s\") pod \"payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.465414 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.465396    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac706e0-2b30-4418-a25d-b5635f50f269-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.478595 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.478575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4h4s\" (UniqueName: \"kubernetes.io/projected/7ac706e0-2b30-4418-a25d-b5635f50f269-kube-api-access-q4h4s\") pod \"payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.511652 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.511630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:45.579563 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.579533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:45.656074 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.655995    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"]
Jul 27 16:55:45.658416 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:55:45.658393    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af73e75_82ef_4a09_ab6c_f7ca5e0d7157.slice/crio-59168de57f8d5f41fb270a32010b93144b6729d1aed540ad41786829125bc053 WatchSource:0}: Error finding container 59168de57f8d5f41fb270a32010b93144b6729d1aed540ad41786829125bc053: Status 404 returned error can't find the container with id 59168de57f8d5f41fb270a32010b93144b6729d1aed540ad41786829125bc053
Jul 27 16:55:45.728769 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:45.728742    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"]
Jul 27 16:55:45.731977 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:55:45.731951    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac706e0_2b30_4418_a25d_b5635f50f269.slice/crio-dfe227d00bc63c3861293691545054a6798def53dd994b39845b42a89be4cc5c WatchSource:0}: Error finding container dfe227d00bc63c3861293691545054a6798def53dd994b39845b42a89be4cc5c: Status 404 returned error can't find the container with id dfe227d00bc63c3861293691545054a6798def53dd994b39845b42a89be4cc5c
Jul 27 16:55:46.239121 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.239085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b" event={"ID":"7ac706e0-2b30-4418-a25d-b5635f50f269","Type":"ContainerStarted","Data":"0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b"}
Jul 27 16:55:46.239333 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.239146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b" event={"ID":"7ac706e0-2b30-4418-a25d-b5635f50f269","Type":"ContainerStarted","Data":"dfe227d00bc63c3861293691545054a6798def53dd994b39845b42a89be4cc5c"}
Jul 27 16:55:46.239333 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.239174    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:46.240842 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.240815    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h" event={"ID":"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157","Type":"ContainerStarted","Data":"1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf"}
Jul 27 16:55:46.240986 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.240857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h" event={"ID":"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157","Type":"ContainerStarted","Data":"59168de57f8d5f41fb270a32010b93144b6729d1aed540ad41786829125bc053"}
Jul 27 16:55:46.240986 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.240883    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:46.263346 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.263302    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b" podStartSLOduration=1.2632690850000001 podStartE2EDuration="1.263269085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:46.261899741 +0000 UTC m=+2346.163040186" watchObservedRunningTime="2026-07-27 16:55:46.263269085 +0000 UTC m=+2346.164409532"
Jul 27 16:55:46.287118 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:46.287071    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h" podStartSLOduration=1.287055578 podStartE2EDuration="1.287055578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:46.28585161 +0000 UTC m=+2346.186992055" watchObservedRunningTime="2026-07-27 16:55:46.287055578 +0000 UTC m=+2346.188196003"
Jul 27 16:55:47.250667 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:47.250632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg" event={"ID":"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd","Type":"ContainerStarted","Data":"d4ea59dc2577a1437fa99e088c325b1e557677afb4da221489535273b405a06a"}
Jul 27 16:55:47.251110 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:47.250953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:47.277298 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:47.277185    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg" podStartSLOduration=1.122614281 podStartE2EDuration="3.27717104s" totalImagesPullingTime="2.154556759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:47.276012068 +0000 UTC m=+2347.177152518" watchObservedRunningTime="2026-07-27 16:55:47.27717104 +0000 UTC m=+2347.178311486"
Jul 27 16:55:48.814755 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:48.814724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:49.182460 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:49.182432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:49.505081 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:49.504978    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:50.440871 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:50.440840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:52.948977 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:52.948922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:53.057946 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.057910    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"]
Jul 27 16:55:53.065356 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.064780    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:53.066941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.066915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"]
Jul 27 16:55:53.067213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.067194    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.070583 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.070559    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5de735-serving-cert\""
Jul 27 16:55:53.150305 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.149132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:53.240661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.240561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6861ed9-9eed-481d-9160-4874db1c6c75-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.240661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.240626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xbq\" (UniqueName: \"kubernetes.io/projected/a6861ed9-9eed-481d-9160-4874db1c6c75-kube-api-access-99xbq\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.240897 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.240702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.263778 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.263751    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:55:53.358303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.355709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.358303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.355804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6861ed9-9eed-481d-9160-4874db1c6c75-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.358303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.355841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xbq\" (UniqueName: \"kubernetes.io/projected/a6861ed9-9eed-481d-9160-4874db1c6c75-kube-api-access-99xbq\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.358303 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:55:53.356241    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-serving-cert: secret "maas-api-e2e-inf-b-5de735-serving-cert" not found
Jul 27 16:55:53.358303 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:55:53.356324    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls podName:a6861ed9-9eed-481d-9160-4874db1c6c75 nodeName:}" failed. No retries permitted until 2026-07-27 16:55:53.856300534 +0000 UTC m=+2353.757440967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls") pod "maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" (UID: "a6861ed9-9eed-481d-9160-4874db1c6c75") : secret "maas-api-e2e-inf-b-5de735-serving-cert" not found
Jul 27 16:55:53.358303 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.357050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6861ed9-9eed-481d-9160-4874db1c6c75-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.369941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.369916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xbq\" (UniqueName: \"kubernetes.io/projected/a6861ed9-9eed-481d-9160-4874db1c6c75-kube-api-access-99xbq\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.591629 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.591545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:53.682736 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.682699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:53.834521 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.834480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:53.860981 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.860881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.863539 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.863513    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls\") pod \"maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:53.979432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:53.979396    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:54.160304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:54.158462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"]
Jul 27 16:55:54.164580 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:55:54.164518    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6861ed9_9eed_481d_9160_4874db1c6c75.slice/crio-aca68efe0a09bb508c8bf59b797af7c27bb18ead9dafa6447473a6f4e0d84656 WatchSource:0}: Error finding container aca68efe0a09bb508c8bf59b797af7c27bb18ead9dafa6447473a6f4e0d84656: Status 404 returned error can't find the container with id aca68efe0a09bb508c8bf59b797af7c27bb18ead9dafa6447473a6f4e0d84656
Jul 27 16:55:54.282637 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:54.282600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" event={"ID":"a6861ed9-9eed-481d-9160-4874db1c6c75","Type":"ContainerStarted","Data":"aca68efe0a09bb508c8bf59b797af7c27bb18ead9dafa6447473a6f4e0d84656"}
Jul 27 16:55:55.289730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:55.289691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" event={"ID":"a6861ed9-9eed-481d-9160-4874db1c6c75","Type":"ContainerStarted","Data":"27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5"}
Jul 27 16:55:55.290224 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:55.289755    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:55:55.312570 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:55.312516    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" podStartSLOduration=1.884587695 podStartE2EDuration="2.312499897s" totalImagesPullingTime="427.912202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:55.310480562 +0000 UTC m=+2355.211621021" watchObservedRunningTime="2026-07-27 16:55:55.312499897 +0000 UTC m=+2355.213640343"
Jul 27 16:55:57.253042 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.253013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:55:57.253501 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.253184    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:55:57.301588 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.301510    2559 generic.go:356] "Generic (PLEG): container finished" podID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerID="9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06" exitCode=0
Jul 27 16:55:57.301713 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.301588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" event={"ID":"08faa6a3-a0a9-4baf-a9d0-d4638a771324","Type":"ContainerDied","Data":"9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"}
Jul 27 16:55:57.302019 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.302002    2559 scope.go:117] "RemoveContainer" containerID="9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"
Jul 27 16:55:57.303057 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.303028    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:55:57.303145 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.303107    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:55:57.303206 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.303151    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:55:57.320569 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:57.320549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:55:58.222028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:58.221978    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:58.308954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:58.308905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" event={"ID":"08faa6a3-a0a9-4baf-a9d0-d4638a771324","Type":"ContainerStarted","Data":"f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d"}
Jul 27 16:55:59.156461 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:59.156416    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:55:59.158103 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:59.158070    2559 patch_prober.go:28] interesting pod/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Jul 27 16:55:59.158344 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:55:59.158316    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Jul 27 16:56:00.156844 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:00.156790    2559 patch_prober.go:28] interesting pod/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Jul 27 16:56:00.157377 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:00.156869    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Jul 27 16:56:01.157384 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:01.157345    2559 patch_prober.go:28] interesting pod/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Jul 27 16:56:01.157897 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:01.157407    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Jul 27 16:56:01.299565 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:01.299536    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:56:02.156476 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:02.156429    2559 patch_prober.go:28] interesting pod/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Jul 27 16:56:02.156690 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:02.156499    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Jul 27 16:56:02.824973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:02.824934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:56:03.160603 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:03.160577    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:56:03.160854 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:03.160833    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:56:03.161656 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:03.161640    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:56:22.868929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:56:22.868890    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:07.110872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:07.110841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:08.008208 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:08.008173    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:18.402371 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.402335    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"]
Jul 27 16:57:18.402815 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.402596    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg" podUID="d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" containerName="maas-api" containerID="cri-o://d4ea59dc2577a1437fa99e088c325b1e557677afb4da221489535273b405a06a" gracePeriod=30
Jul 27 16:57:18.429389 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.429365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:18.592216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.592181    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"]
Jul 27 16:57:18.611103 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.611065    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"]
Jul 27 16:57:18.611373 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.611320    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b" podUID="7ac706e0-2b30-4418-a25d-b5635f50f269" containerName="payload-pre-processing" containerID="cri-o://0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b" gracePeriod=30
Jul 27 16:57:18.624757 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.624729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:18.653983 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.653914    2559 generic.go:356] "Generic (PLEG): container finished" podID="d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" containerID="d4ea59dc2577a1437fa99e088c325b1e557677afb4da221489535273b405a06a" exitCode=0
Jul 27 16:57:18.654108 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.653985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg" event={"ID":"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd","Type":"ContainerDied","Data":"d4ea59dc2577a1437fa99e088c325b1e557677afb4da221489535273b405a06a"}
Jul 27 16:57:18.654108 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.654061    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h" podUID="7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" containerName="payload-processing" containerID="cri-o://1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf" gracePeriod=30
Jul 27 16:57:18.723762 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.723718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:18.765678 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.765652    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:57:18.883731 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.883701    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-openshift-service-ca\") pod \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") "
Jul 27 16:57:18.883903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.883738    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dvp6\" (UniqueName: \"kubernetes.io/projected/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-kube-api-access-4dvp6\") pod \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") "
Jul 27 16:57:18.883903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.883806    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-maas-api-tls\") pod \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\" (UID: \"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd\") "
Jul 27 16:57:18.884138 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.884111    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" (UID: "d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:57:18.887135 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.887105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" (UID: "d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:57:18.888586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.888563    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-kube-api-access-4dvp6" (OuterVolumeSpecName: "kube-api-access-4dvp6") pod "d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" (UID: "d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd"). InnerVolumeSpecName "kube-api-access-4dvp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:18.914225 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.914200    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:57:18.924631 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.924608    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:57:18.984545 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.984516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac706e0-2b30-4418-a25d-b5635f50f269-plugins-config-volume\") pod \"7ac706e0-2b30-4418-a25d-b5635f50f269\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") "
Jul 27 16:57:18.984707 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.984579    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4h4s\" (UniqueName: \"kubernetes.io/projected/7ac706e0-2b30-4418-a25d-b5635f50f269-kube-api-access-q4h4s\") pod \"7ac706e0-2b30-4418-a25d-b5635f50f269\" (UID: \"7ac706e0-2b30-4418-a25d-b5635f50f269\") "
Jul 27 16:57:18.984757 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.984747    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:18.984796 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.984760    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:18.984796 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.984771    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dvp6\" (UniqueName: \"kubernetes.io/projected/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd-kube-api-access-4dvp6\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:18.984876 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.984855    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac706e0-2b30-4418-a25d-b5635f50f269-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ac706e0-2b30-4418-a25d-b5635f50f269" (UID: "7ac706e0-2b30-4418-a25d-b5635f50f269"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:57:18.986642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:18.986615    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac706e0-2b30-4418-a25d-b5635f50f269-kube-api-access-q4h4s" (OuterVolumeSpecName: "kube-api-access-q4h4s") pod "7ac706e0-2b30-4418-a25d-b5635f50f269" (UID: "7ac706e0-2b30-4418-a25d-b5635f50f269"). InnerVolumeSpecName "kube-api-access-q4h4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:19.085211 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.085177    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2hwq\" (UniqueName: \"kubernetes.io/projected/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-kube-api-access-c2hwq\") pod \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") "
Jul 27 16:57:19.085387 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.085235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-plugins-config-volume\") pod \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\" (UID: \"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157\") "
Jul 27 16:57:19.085497 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.085479    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4h4s\" (UniqueName: \"kubernetes.io/projected/7ac706e0-2b30-4418-a25d-b5635f50f269-kube-api-access-q4h4s\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:19.085530 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.085503    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ac706e0-2b30-4418-a25d-b5635f50f269-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:19.085598 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.085576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" (UID: "7af73e75-82ef-4a09-ab6c-f7ca5e0d7157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:57:19.087221 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.087199    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-kube-api-access-c2hwq" (OuterVolumeSpecName: "kube-api-access-c2hwq") pod "7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" (UID: "7af73e75-82ef-4a09-ab6c-f7ca5e0d7157"). InnerVolumeSpecName "kube-api-access-c2hwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:19.186040 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.185962    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2hwq\" (UniqueName: \"kubernetes.io/projected/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-kube-api-access-c2hwq\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:19.186040 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.185992    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:19.660021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.659990    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"
Jul 27 16:57:19.660021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.659998    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg" event={"ID":"d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd","Type":"ContainerDied","Data":"f2ba9b7d42d90eaf927c39e2c9af6c8f3bf0d5b4e86803a7b509fec9a064db3d"}
Jul 27 16:57:19.660577 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.660087    2559 scope.go:117] "RemoveContainer" containerID="d4ea59dc2577a1437fa99e088c325b1e557677afb4da221489535273b405a06a"
Jul 27 16:57:19.661412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.661383    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ac706e0-2b30-4418-a25d-b5635f50f269" containerID="0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b" exitCode=0
Jul 27 16:57:19.661484 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.661427    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"
Jul 27 16:57:19.661484 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.661460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b" event={"ID":"7ac706e0-2b30-4418-a25d-b5635f50f269","Type":"ContainerDied","Data":"0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b"}
Jul 27 16:57:19.661597 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.661498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b" event={"ID":"7ac706e0-2b30-4418-a25d-b5635f50f269","Type":"ContainerDied","Data":"dfe227d00bc63c3861293691545054a6798def53dd994b39845b42a89be4cc5c"}
Jul 27 16:57:19.662885 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.662860    2559 generic.go:356] "Generic (PLEG): container finished" podID="7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" containerID="1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf" exitCode=0
Jul 27 16:57:19.663008 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.662890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h" event={"ID":"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157","Type":"ContainerDied","Data":"1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf"}
Jul 27 16:57:19.663008 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.662904    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"
Jul 27 16:57:19.663008 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.662940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h" event={"ID":"7af73e75-82ef-4a09-ab6c-f7ca5e0d7157","Type":"ContainerDied","Data":"59168de57f8d5f41fb270a32010b93144b6729d1aed540ad41786829125bc053"}
Jul 27 16:57:19.669240 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.669220    2559 scope.go:117] "RemoveContainer" containerID="0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b"
Jul 27 16:57:19.677998 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.677983    2559 scope.go:117] "RemoveContainer" containerID="0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b"
Jul 27 16:57:19.678246 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:19.678229    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b\": container with ID starting with 0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b not found: ID does not exist" containerID="0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b"
Jul 27 16:57:19.678323 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.678257    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b"} err="failed to get container status \"0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b\": rpc error: code = NotFound desc = could not find container \"0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b\": container with ID starting with 0c521304a2ac09b461b5d7e1207c91c9ee1a331624b5d9bf582da36c22ddbc9b not found: ID does not exist"
Jul 27 16:57:19.678323 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.678273    2559 scope.go:117] "RemoveContainer" containerID="1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf"
Jul 27 16:57:19.687525 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.687424    2559 scope.go:117] "RemoveContainer" containerID="1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf"
Jul 27 16:57:19.687836 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:19.687815    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf\": container with ID starting with 1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf not found: ID does not exist" containerID="1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf"
Jul 27 16:57:19.687900 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.687842    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf"} err="failed to get container status \"1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf\": rpc error: code = NotFound desc = could not find container \"1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf\": container with ID starting with 1dcf6a55beaecd977327e1a92773e9f049980dbf409bbce8408c2d19c1d6cbbf not found: ID does not exist"
Jul 27 16:57:19.688312 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.688293    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"]
Jul 27 16:57:19.692006 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.691986    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23e152-659b8596fb-56m4b"]
Jul 27 16:57:19.703502 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.703481    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"]
Jul 27 16:57:19.708388 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.708368    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23e152-759f974b-nv48h"]
Jul 27 16:57:19.719511 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.719490    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"]
Jul 27 16:57:19.721687 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:19.721664    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-23e152-675f696dd5-xbffg"]
Jul 27 16:57:20.734684 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:20.734655    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac706e0-2b30-4418-a25d-b5635f50f269" path="/var/lib/kubelet/pods/7ac706e0-2b30-4418-a25d-b5635f50f269/volumes"
Jul 27 16:57:20.735053 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:20.734984    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" path="/var/lib/kubelet/pods/7af73e75-82ef-4a09-ab6c-f7ca5e0d7157/volumes"
Jul 27 16:57:20.735331 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:20.735317    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" path="/var/lib/kubelet/pods/d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd/volumes"
Jul 27 16:57:23.237572 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:23.237536    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:23.630141 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:23.630054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:29.893029 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:29.892995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:29.973586 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:29.973553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:30.165756 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:30.165680    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"]
Jul 27 16:57:30.165978 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:30.165951    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy" containerID="cri-o://f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d" gracePeriod=30
Jul 27 16:57:30.183239 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:30.183212    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:30.421139 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:30.421060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:33.163686 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:33.163649    2559 patch_prober.go:28] interesting pod/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 16:57:33.164046 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:33.163721    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 16:57:35.413925 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.413899    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:57:35.421795 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421766    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-envoy\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.421915 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421821    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-token\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.421915 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421853    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g64s6\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-kube-api-access-g64s6\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.421915 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421882    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-socket\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.421915 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421908    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-podinfo\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.422116 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421947    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-certs\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.422116 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.421978    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istiod-ca-cert\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.422240 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422215    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-data\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.422317 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422249    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:35.422317 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422259    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-credential-socket\") pod \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\" (UID: \"08faa6a3-a0a9-4baf-a9d0-d4638a771324\") "
Jul 27 16:57:35.422426 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422381    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:57:35.422506 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422470    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:35.422709 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422640    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.422709 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422658    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.422709 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422672    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.422984 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.422954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:35.423314 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.423275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-data" (OuterVolumeSpecName: "istio-data") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:35.424092 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.424060    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:35.424169 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.424146    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-kube-api-access-g64s6" (OuterVolumeSpecName: "kube-api-access-g64s6") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "kube-api-access-g64s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:35.424608 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.424355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-token" (OuterVolumeSpecName: "istio-token") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:35.424747 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.424366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08faa6a3-a0a9-4baf-a9d0-d4638a771324" (UID: "08faa6a3-a0a9-4baf-a9d0-d4638a771324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:57:35.522991 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.522957    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.522991 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.522985    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.522991 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.522994    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g64s6\" (UniqueName: \"kubernetes.io/projected/08faa6a3-a0a9-4baf-a9d0-d4638a771324-kube-api-access-g64s6\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.523213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.523004    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.523213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.523014    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.523213 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.523022    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08faa6a3-a0a9-4baf-a9d0-d4638a771324-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:35.736579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.736491    2559 generic.go:356] "Generic (PLEG): container finished" podID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerID="f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d" exitCode=0
Jul 27 16:57:35.736579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.736549    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"
Jul 27 16:57:35.736753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.736579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" event={"ID":"08faa6a3-a0a9-4baf-a9d0-d4638a771324","Type":"ContainerDied","Data":"f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d"}
Jul 27 16:57:35.736753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.736633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz" event={"ID":"08faa6a3-a0a9-4baf-a9d0-d4638a771324","Type":"ContainerDied","Data":"055483c5433ccf0b75acc08aad10180bb8f10b1d4b543dfb69f93da538cf26a9"}
Jul 27 16:57:35.736753 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.736656    2559 scope.go:117] "RemoveContainer" containerID="f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d"
Jul 27 16:57:35.745135 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.745120    2559 scope.go:117] "RemoveContainer" containerID="9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"
Jul 27 16:57:35.752449 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.752434    2559 scope.go:117] "RemoveContainer" containerID="f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d"
Jul 27 16:57:35.752667 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:35.752650    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d\": container with ID starting with f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d not found: ID does not exist" containerID="f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d"
Jul 27 16:57:35.752708 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.752675    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d"} err="failed to get container status \"f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d\": rpc error: code = NotFound desc = could not find container \"f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d\": container with ID starting with f7be7d5c4c801a9e5b93485db1729179349d7aaf9d28b4d4fb35d22f6459994d not found: ID does not exist"
Jul 27 16:57:35.752708 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.752691    2559 scope.go:117] "RemoveContainer" containerID="9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"
Jul 27 16:57:35.752888 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:35.752873    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06\": container with ID starting with 9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06 not found: ID does not exist" containerID="9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"
Jul 27 16:57:35.752924 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.752892    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06"} err="failed to get container status \"9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06\": rpc error: code = NotFound desc = could not find container \"9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06\": container with ID starting with 9f49cf28a1e2dc0ef34e6286d9b0af6fd7437ec735e583e65ea9f8e24a4fbb06 not found: ID does not exist"
Jul 27 16:57:35.759048 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.759028    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"]
Jul 27 16:57:35.763801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:35.763781    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-23e152-openshift-default-7f6d795df8-vbnbz"]
Jul 27 16:57:36.735566 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:36.735532    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" path="/var/lib/kubelet/pods/08faa6a3-a0a9-4baf-a9d0-d4638a771324/volumes"
Jul 27 16:57:40.971640 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:40.971610    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"]
Jul 27 16:57:40.972038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:40.971872    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" podUID="a6861ed9-9eed-481d-9160-4874db1c6c75" containerName="maas-api" containerID="cri-o://27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5" gracePeriod=30
Jul 27 16:57:41.001412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.001372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:41.088684 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.088597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:41.165676 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.165640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:41.295312 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.295256    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" podUID="a6861ed9-9eed-481d-9160-4874db1c6c75" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.54:8443/health\": dial tcp 10.132.0.54:8443: connect: connection refused"
Jul 27 16:57:41.412398 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.412371    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:57:41.481870 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.481835    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99xbq\" (UniqueName: \"kubernetes.io/projected/a6861ed9-9eed-481d-9160-4874db1c6c75-kube-api-access-99xbq\") pod \"a6861ed9-9eed-481d-9160-4874db1c6c75\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") "
Jul 27 16:57:41.482034 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.481878    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls\") pod \"a6861ed9-9eed-481d-9160-4874db1c6c75\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") "
Jul 27 16:57:41.482034 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.481904    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6861ed9-9eed-481d-9160-4874db1c6c75-openshift-service-ca\") pod \"a6861ed9-9eed-481d-9160-4874db1c6c75\" (UID: \"a6861ed9-9eed-481d-9160-4874db1c6c75\") "
Jul 27 16:57:41.482330 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.482304    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6861ed9-9eed-481d-9160-4874db1c6c75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6861ed9-9eed-481d-9160-4874db1c6c75" (UID: "a6861ed9-9eed-481d-9160-4874db1c6c75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:57:41.483900 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.483880    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6861ed9-9eed-481d-9160-4874db1c6c75" (UID: "a6861ed9-9eed-481d-9160-4874db1c6c75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:57:41.483988 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.483969    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6861ed9-9eed-481d-9160-4874db1c6c75-kube-api-access-99xbq" (OuterVolumeSpecName: "kube-api-access-99xbq") pod "a6861ed9-9eed-481d-9160-4874db1c6c75" (UID: "a6861ed9-9eed-481d-9160-4874db1c6c75"). InnerVolumeSpecName "kube-api-access-99xbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:41.582550 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.582517    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xbq\" (UniqueName: \"kubernetes.io/projected/a6861ed9-9eed-481d-9160-4874db1c6c75-kube-api-access-99xbq\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:41.582550 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.582545    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6861ed9-9eed-481d-9160-4874db1c6c75-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:41.582550 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.582555    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6861ed9-9eed-481d-9160-4874db1c6c75-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:57:41.761851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.761758    2559 generic.go:356] "Generic (PLEG): container finished" podID="a6861ed9-9eed-481d-9160-4874db1c6c75" containerID="27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5" exitCode=0
Jul 27 16:57:41.761851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.761810    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"
Jul 27 16:57:41.761851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.761840    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" event={"ID":"a6861ed9-9eed-481d-9160-4874db1c6c75","Type":"ContainerDied","Data":"27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5"}
Jul 27 16:57:41.762061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.761877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj" event={"ID":"a6861ed9-9eed-481d-9160-4874db1c6c75","Type":"ContainerDied","Data":"aca68efe0a09bb508c8bf59b797af7c27bb18ead9dafa6447473a6f4e0d84656"}
Jul 27 16:57:41.762061 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.761899    2559 scope.go:117] "RemoveContainer" containerID="27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5"
Jul 27 16:57:41.769986 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.769970    2559 scope.go:117] "RemoveContainer" containerID="27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5"
Jul 27 16:57:41.770244 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:41.770223    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5\": container with ID starting with 27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5 not found: ID does not exist" containerID="27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5"
Jul 27 16:57:41.770320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.770251    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5"} err="failed to get container status \"27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5\": rpc error: code = NotFound desc = could not find container \"27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5\": container with ID starting with 27627d89882f02b6b382cf75136bb247aaaf1d0a0c4f19a891697289998f4ad5 not found: ID does not exist"
Jul 27 16:57:41.786128 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.786100    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"]
Jul 27 16:57:41.790895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:41.790872    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5de735-dd5999f8f-xc8xj"]
Jul 27 16:57:42.735233 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:42.735200    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6861ed9-9eed-481d-9160-4874db1c6c75" path="/var/lib/kubelet/pods/a6861ed9-9eed-481d-9160-4874db1c6c75/volumes"
Jul 27 16:57:45.630370 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:45.630336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:46.035191 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:46.035160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:52.322960 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:52.322922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:52.423614 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:52.423587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:52.626414 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:52.626331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:52.870294 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:52.870259    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:53.251336 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.251302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:53.547562 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547483    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"]
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547793    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6861ed9-9eed-481d-9160-4874db1c6c75" containerName="maas-api"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547805    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6861ed9-9eed-481d-9160-4874db1c6c75" containerName="maas-api"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547817    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" containerName="payload-processing"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547822    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" containerName="payload-processing"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547834    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547840    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547849    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547854    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547864    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac706e0-2b30-4418-a25d-b5635f50f269" containerName="payload-pre-processing"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547869    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac706e0-2b30-4418-a25d-b5635f50f269" containerName="payload-pre-processing"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547878    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" containerName="maas-api"
Jul 27 16:57:53.547895 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547883    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" containerName="maas-api"
Jul 27 16:57:53.548238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547926    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac706e0-2b30-4418-a25d-b5635f50f269" containerName="payload-pre-processing"
Jul 27 16:57:53.548238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547932    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af73e75-82ef-4a09-ab6c-f7ca5e0d7157" containerName="payload-processing"
Jul 27 16:57:53.548238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547937    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26b937a-1d7c-4e07-b6d7-b0ef83f43bfd" containerName="maas-api"
Jul 27 16:57:53.548238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547942    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy"
Jul 27 16:57:53.548238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547949    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08faa6a3-a0a9-4baf-a9d0-d4638a771324" containerName="istio-proxy"
Jul 27 16:57:53.548238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.547954    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6861ed9-9eed-481d-9160-4874db1c6c75" containerName="maas-api"
Jul 27 16:57:53.550774 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.550755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.554026 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.554005    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6dee51-openshift-default-dockercfg-m4d2k\""
Jul 27 16:57:53.564080 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.564060    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"]
Jul 27 16:57:53.580475 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.580443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:53.683822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-data\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.683987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-envoy\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.683987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-podinfo\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.683987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-certs\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.683987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-token\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.683987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-socket\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.684155 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.683991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istiod-ca-cert\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.684155 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.684065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbp5f\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-kube-api-access-cbp5f\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.684155 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.684109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-credential-socket\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785394 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-credential-socket\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-data\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-envoy\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-podinfo\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-certs\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-token\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-socket\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istiod-ca-cert\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbp5f\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-kube-api-access-cbp5f\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-data\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.785959 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.785861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-credential-socket\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.786067 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.786015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-socket\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.786067 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.786022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-certs\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.786412 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.786395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istiod-ca-cert\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.787852 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.787830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-envoy\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.788123 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.788101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-podinfo\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.802915 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.802854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-token\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.803160 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.803138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbp5f\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-kube-api-access-cbp5f\") pod \"e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") " pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.860588 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.860555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:53.872559 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:53.872530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:54.009475 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.009451    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"]
Jul 27 16:57:54.011863 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:57:54.011831    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68e96da_52bb_4c23_bad5_2fb6f40cdb85.slice/crio-369d34b7919f5b9c07247a9184647a5a04cb52d4aab6ab76492a12c62fd72a1a WatchSource:0}: Error finding container 369d34b7919f5b9c07247a9184647a5a04cb52d4aab6ab76492a12c62fd72a1a: Status 404 returned error can't find the container with id 369d34b7919f5b9c07247a9184647a5a04cb52d4aab6ab76492a12c62fd72a1a
Jul 27 16:57:54.013779 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.013742    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:57:54.013872 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.013844    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:57:54.013929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.013887    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:57:54.815582 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.815549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" event={"ID":"c68e96da-52bb-4c23-bad5-2fb6f40cdb85","Type":"ContainerStarted","Data":"be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"}
Jul 27 16:57:54.815949 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.815599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" event={"ID":"c68e96da-52bb-4c23-bad5-2fb6f40cdb85","Type":"ContainerStarted","Data":"369d34b7919f5b9c07247a9184647a5a04cb52d4aab6ab76492a12c62fd72a1a"}
Jul 27 16:57:54.842662 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.842608    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podStartSLOduration=1.842593742 podStartE2EDuration="1.842593742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:54.841104429 +0000 UTC m=+2474.742244878" watchObservedRunningTime="2026-07-27 16:57:54.842593742 +0000 UTC m=+2474.743734188"
Jul 27 16:57:54.861289 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:54.861262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:55.865774 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:55.865736    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:56.824658 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:56.824629    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:56.825536 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:56.825518    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:57:56.886696 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:56.886656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:56.972608 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:56.972573    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:57.136851 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.136777    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"]
Jul 27 16:57:57.140053 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.140033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.143718 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.143693    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:57:57.143846 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.143704    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6dee51-serving-cert\""
Jul 27 16:57:57.143846 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.143755    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:57:57.144163 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.144150    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:57:57.160399 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.160370    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"]
Jul 27 16:57:57.193024 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.192996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:57.218973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.218753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d83af-38c3-4bd1-914d-794a2e9da75c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.218973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.218811    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.218973 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.218884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dx2\" (UniqueName: \"kubernetes.io/projected/7b0d83af-38c3-4bd1-914d-794a2e9da75c-kube-api-access-99dx2\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.319759 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.319729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dx2\" (UniqueName: \"kubernetes.io/projected/7b0d83af-38c3-4bd1-914d-794a2e9da75c-kube-api-access-99dx2\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.319945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.319794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d83af-38c3-4bd1-914d-794a2e9da75c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.319945 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.319818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.320012 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:57.319991    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-serving-cert: secret "maas-api-e2e-api-a-6dee51-serving-cert" not found
Jul 27 16:57:57.320054 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:57:57.320045    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls podName:7b0d83af-38c3-4bd1-914d-794a2e9da75c nodeName:}" failed. No retries permitted until 2026-07-27 16:57:57.820030646 +0000 UTC m=+2477.721171071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls") pod "maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" (UID: "7b0d83af-38c3-4bd1-914d-794a2e9da75c") : secret "maas-api-e2e-api-a-6dee51-serving-cert" not found
Jul 27 16:57:57.320457 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.320439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d83af-38c3-4bd1-914d-794a2e9da75c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.349080 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.349054    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dx2\" (UniqueName: \"kubernetes.io/projected/7b0d83af-38c3-4bd1-914d-794a2e9da75c-kube-api-access-99dx2\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.824951 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.824915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.827848 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.827821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls\") pod \"maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:57.971094 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.971057    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"]
Jul 27 16:57:57.976371 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.976347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:57.979324 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.979303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6dee51-dockercfg-5f2cd\""
Jul 27 16:57:57.979825 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.979808    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6dee51\""
Jul 27 16:57:57.988030 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:57.988009    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"]
Jul 27 16:57:58.049995 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.049969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:58.101464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.101381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:58.127524 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.127493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8bw\" (UniqueName: \"kubernetes.io/projected/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-kube-api-access-nc8bw\") pod \"payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.127705 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.127554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.205216 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.205158    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"]
Jul 27 16:57:58.207797 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:57:58.207767    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0d83af_38c3_4bd1_914d_794a2e9da75c.slice/crio-850b7238f18c0cfcc7708e67d807f3fcdaf5dc3b1df8eb07e58a603fa19138a8 WatchSource:0}: Error finding container 850b7238f18c0cfcc7708e67d807f3fcdaf5dc3b1df8eb07e58a603fa19138a8: Status 404 returned error can't find the container with id 850b7238f18c0cfcc7708e67d807f3fcdaf5dc3b1df8eb07e58a603fa19138a8
Jul 27 16:57:58.209611 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.209581    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:57:58.211829 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.211797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:58.228322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.228273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8bw\" (UniqueName: \"kubernetes.io/projected/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-kube-api-access-nc8bw\") pod \"payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.228443 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.228379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.229069 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.229050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.240868 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.240846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8bw\" (UniqueName: \"kubernetes.io/projected/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-kube-api-access-nc8bw\") pod \"payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") " pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.286401 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.286377    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.419196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.419164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"]
Jul 27 16:57:58.423239 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:57:58.423215    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a268b4c_d6ad_4bbc_8991_c5ee4ccb0db8.slice/crio-0b44a3340acb7a80c325cf13e12ea361b79cac5a42939d42db7eaacdc04671e5 WatchSource:0}: Error finding container 0b44a3340acb7a80c325cf13e12ea361b79cac5a42939d42db7eaacdc04671e5: Status 404 returned error can't find the container with id 0b44a3340acb7a80c325cf13e12ea361b79cac5a42939d42db7eaacdc04671e5
Jul 27 16:57:58.835142 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.835108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g" event={"ID":"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8","Type":"ContainerStarted","Data":"45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e"}
Jul 27 16:57:58.835359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.835170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g" event={"ID":"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8","Type":"ContainerStarted","Data":"0b44a3340acb7a80c325cf13e12ea361b79cac5a42939d42db7eaacdc04671e5"}
Jul 27 16:57:58.835359 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.835200    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:57:58.836272 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:58.836252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" event={"ID":"7b0d83af-38c3-4bd1-914d-794a2e9da75c","Type":"ContainerStarted","Data":"850b7238f18c0cfcc7708e67d807f3fcdaf5dc3b1df8eb07e58a603fa19138a8"}
Jul 27 16:57:59.842612 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:59.842512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" event={"ID":"7b0d83af-38c3-4bd1-914d-794a2e9da75c","Type":"ContainerStarted","Data":"80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2"}
Jul 27 16:57:59.843035 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:59.842672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:57:59.860210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:59.860168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" podStartSLOduration=1.623242967 podStartE2EDuration="2.860153004s" totalImagesPullingTime="1.236910037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:59.859827772 +0000 UTC m=+2479.760968220" watchObservedRunningTime="2026-07-27 16:57:59.860153004 +0000 UTC m=+2479.761293451"
Jul 27 16:57:59.860838 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:57:59.860809    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g" podStartSLOduration=2.8607995710000003 podStartE2EDuration="2.860799571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:58.868018637 +0000 UTC m=+2478.769159083" watchObservedRunningTime="2026-07-27 16:57:59.860799571 +0000 UTC m=+2479.761940018"
Jul 27 16:58:01.465047 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:01.465014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:01.959102 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:01.959070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:02.297604 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:02.297523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:02.619004 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:02.618928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:05.536463 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.536425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:05.659050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.659009    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"]
Jul 27 16:58:05.662903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.662884    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.666187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.666170    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-30f382-serving-cert\""
Jul 27 16:58:05.675161 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.675137    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"]
Jul 27 16:58:05.753519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.753488    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:05.790827 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.790746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac93956-3b34-4b0d-8549-067dc25a494a-maas-api-tls\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.790975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.790842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrzx\" (UniqueName: \"kubernetes.io/projected/eac93956-3b34-4b0d-8549-067dc25a494a-kube-api-access-xzrzx\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.790975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.790870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac93956-3b34-4b0d-8549-067dc25a494a-openshift-service-ca\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.856785 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.856734    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:05.861621 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.861589    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:58:05.891515 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.891481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrzx\" (UniqueName: \"kubernetes.io/projected/eac93956-3b34-4b0d-8549-067dc25a494a-kube-api-access-xzrzx\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.891515 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.891517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac93956-3b34-4b0d-8549-067dc25a494a-openshift-service-ca\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.891778 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.891596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac93956-3b34-4b0d-8549-067dc25a494a-maas-api-tls\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.892206 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.892187    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac93956-3b34-4b0d-8549-067dc25a494a-openshift-service-ca\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.894095 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.894078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac93956-3b34-4b0d-8549-067dc25a494a-maas-api-tls\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.907953 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.907922    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrzx\" (UniqueName: \"kubernetes.io/projected/eac93956-3b34-4b0d-8549-067dc25a494a-kube-api-access-xzrzx\") pod \"maas-api-e2e-api-b-30f382-548cd5747f-7gxsh\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:05.961321 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.956639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:05.972663 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:05.972630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:06.128163 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.128141    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"]
Jul 27 16:58:06.130772 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:58:06.130734    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac93956_3b34_4b0d_8549_067dc25a494a.slice/crio-82b851d5e4a5c81ca167c94b583c42d88748bf7e1bbd0bed44b941552636c05d WatchSource:0}: Error finding container 82b851d5e4a5c81ca167c94b583c42d88748bf7e1bbd0bed44b941552636c05d: Status 404 returned error can't find the container with id 82b851d5e4a5c81ca167c94b583c42d88748bf7e1bbd0bed44b941552636c05d
Jul 27 16:58:06.277617 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.277579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:06.379920 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.379842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:06.493701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.493666    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"]
Jul 27 16:58:06.498173 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.498156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.500957 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.500935    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-30f382\""
Jul 27 16:58:06.501093 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.500977    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-30f382-dockercfg-7994d\""
Jul 27 16:58:06.504519 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.504495    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"]
Jul 27 16:58:06.598239 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.598201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjbj\" (UniqueName: \"kubernetes.io/projected/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-kube-api-access-rjjbj\") pod \"payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.598617 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.598248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.698787 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.698763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjbj\" (UniqueName: \"kubernetes.io/projected/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-kube-api-access-rjjbj\") pod \"payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.698887 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.698800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.699491 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.699473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.708021 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.708002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjbj\" (UniqueName: \"kubernetes.io/projected/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-kube-api-access-rjjbj\") pod \"payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.810226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.810187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:06.875465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.875275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh" event={"ID":"eac93956-3b34-4b0d-8549-067dc25a494a","Type":"ContainerStarted","Data":"0852c7e6887ecbc8a1642594f14b84145861ea10cb980c8ea373563a754e50cd"}
Jul 27 16:58:06.875465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.875346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh" event={"ID":"eac93956-3b34-4b0d-8549-067dc25a494a","Type":"ContainerStarted","Data":"82b851d5e4a5c81ca167c94b583c42d88748bf7e1bbd0bed44b941552636c05d"}
Jul 27 16:58:06.875465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.875388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:06.896079 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.896022    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh" podStartSLOduration=1.396607199 podStartE2EDuration="1.89600428s" totalImagesPullingTime="499.397081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:06.895018306 +0000 UTC m=+2486.796158754" watchObservedRunningTime="2026-07-27 16:58:06.89600428 +0000 UTC m=+2486.797144781"
Jul 27 16:58:06.994805 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:06.994613    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"]
Jul 27 16:58:06.997578 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:58:06.997554    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefbbd03_1141_4aae_a1ab_43ee67ff1fb0.slice/crio-6ba7e0198bfdf03b7d79d3e1aa13804cba34513668c0719b777a8623c5d072c7 WatchSource:0}: Error finding container 6ba7e0198bfdf03b7d79d3e1aa13804cba34513668c0719b777a8623c5d072c7: Status 404 returned error can't find the container with id 6ba7e0198bfdf03b7d79d3e1aa13804cba34513668c0719b777a8623c5d072c7
Jul 27 16:58:07.882987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:07.882948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q" event={"ID":"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0","Type":"ContainerStarted","Data":"12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39"}
Jul 27 16:58:07.883505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:07.883009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q" event={"ID":"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0","Type":"ContainerStarted","Data":"6ba7e0198bfdf03b7d79d3e1aa13804cba34513668c0719b777a8623c5d072c7"}
Jul 27 16:58:07.883505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:07.883171    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:07.901038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:07.900993    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q" podStartSLOduration=1.900978745 podStartE2EDuration="1.900978745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:07.900330995 +0000 UTC m=+2487.801471442" watchObservedRunningTime="2026-07-27 16:58:07.900978745 +0000 UTC m=+2487.802119190"
Jul 27 16:58:09.844815 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:09.844785    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:58:10.101525 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:10.101441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:11.481336 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:11.481296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:12.547322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.546998    2559 patch_prober.go:28] interesting pod/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:58774->10.132.0.55:15021: read: connection reset by peer" start-of-body=
Jul 27 16:58:12.547322 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.547052    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:58774->10.132.0.55:15021: read: connection reset by peer"
Jul 27 16:58:12.888536 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.888441    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:58:12.907301 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.907252    2559 generic.go:356] "Generic (PLEG): container finished" podID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerID="be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee" exitCode=0
Jul 27 16:58:12.907451 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.907326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" event={"ID":"c68e96da-52bb-4c23-bad5-2fb6f40cdb85","Type":"ContainerDied","Data":"be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"}
Jul 27 16:58:12.907673 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.907659    2559 scope.go:117] "RemoveContainer" containerID="be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"
Jul 27 16:58:12.908728 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.908702    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:58:12.908787 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.908769    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:58:12.908822 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:12.908798    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:58:13.913610 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:13.913577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" event={"ID":"c68e96da-52bb-4c23-bad5-2fb6f40cdb85","Type":"ContainerStarted","Data":"a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96"}
Jul 27 16:58:14.860975 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:14.860930    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:58:14.861214 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:14.861190    2559 patch_prober.go:28] interesting pod/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 27 16:58:14.861258 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:14.861241    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 27 16:58:15.861754 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:15.861719    2559 patch_prober.go:28] interesting pod/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 27 16:58:15.862137 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:15.861792    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 27 16:58:16.861714 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:16.861676    2559 patch_prober.go:28] interesting pod/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 27 16:58:16.861903 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:16.861737    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 27 16:58:17.864954 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:17.864924    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:58:17.865336 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:17.865184    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:58:17.865775 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:17.865751    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:58:18.889719 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:18.889691    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:58:34.054950 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.054759    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"]
Jul 27 16:58:34.055627 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.055593    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" podUID="7b0d83af-38c3-4bd1-914d-794a2e9da75c" containerName="maas-api" containerID="cri-o://80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2" gracePeriod=30
Jul 27 16:58:34.078835 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.078810    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:34.108521 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.108486    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"]
Jul 27 16:58:34.108801 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.108767    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g" podUID="3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" containerName="payload-processing" containerID="cri-o://45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e" gracePeriod=30
Jul 27 16:58:34.179130 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.179091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:34.281238 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.281170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:34.346234 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.346212    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:58:34.376051 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.376024    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:58:34.441395 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441368    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-plugins-config-volume\") pod \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") "
Jul 27 16:58:34.441557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441412    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d83af-38c3-4bd1-914d-794a2e9da75c-openshift-service-ca\") pod \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") "
Jul 27 16:58:34.441557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99dx2\" (UniqueName: \"kubernetes.io/projected/7b0d83af-38c3-4bd1-914d-794a2e9da75c-kube-api-access-99dx2\") pod \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") "
Jul 27 16:58:34.441557 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441504    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls\") pod \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\" (UID: \"7b0d83af-38c3-4bd1-914d-794a2e9da75c\") "
Jul 27 16:58:34.441727 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441562    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc8bw\" (UniqueName: \"kubernetes.io/projected/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-kube-api-access-nc8bw\") pod \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\" (UID: \"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8\") "
Jul 27 16:58:34.441843 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441807    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" (UID: "3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:58:34.441916 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.441814    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0d83af-38c3-4bd1-914d-794a2e9da75c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b0d83af-38c3-4bd1-914d-794a2e9da75c" (UID: "7b0d83af-38c3-4bd1-914d-794a2e9da75c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:58:34.443609 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.443584    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b0d83af-38c3-4bd1-914d-794a2e9da75c" (UID: "7b0d83af-38c3-4bd1-914d-794a2e9da75c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:58:34.443681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.443605    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0d83af-38c3-4bd1-914d-794a2e9da75c-kube-api-access-99dx2" (OuterVolumeSpecName: "kube-api-access-99dx2") pod "7b0d83af-38c3-4bd1-914d-794a2e9da75c" (UID: "7b0d83af-38c3-4bd1-914d-794a2e9da75c"). InnerVolumeSpecName "kube-api-access-99dx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:34.443681 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.443629    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-kube-api-access-nc8bw" (OuterVolumeSpecName: "kube-api-access-nc8bw") pod "3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" (UID: "3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8"). InnerVolumeSpecName "kube-api-access-nc8bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:34.542262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.542235    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc8bw\" (UniqueName: \"kubernetes.io/projected/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-kube-api-access-nc8bw\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:34.542262 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.542259    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:34.542449 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.542269    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b0d83af-38c3-4bd1-914d-794a2e9da75c-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:34.542449 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.542299    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dx2\" (UniqueName: \"kubernetes.io/projected/7b0d83af-38c3-4bd1-914d-794a2e9da75c-kube-api-access-99dx2\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:34.542449 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:34.542308    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b0d83af-38c3-4bd1-914d-794a2e9da75c-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:35.007320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.007220    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b0d83af-38c3-4bd1-914d-794a2e9da75c" containerID="80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2" exitCode=0
Jul 27 16:58:35.007320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.007263    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"
Jul 27 16:58:35.007320 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.007305    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" event={"ID":"7b0d83af-38c3-4bd1-914d-794a2e9da75c","Type":"ContainerDied","Data":"80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2"}
Jul 27 16:58:35.007579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.007359    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8" event={"ID":"7b0d83af-38c3-4bd1-914d-794a2e9da75c","Type":"ContainerDied","Data":"850b7238f18c0cfcc7708e67d807f3fcdaf5dc3b1df8eb07e58a603fa19138a8"}
Jul 27 16:58:35.007579 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.007387    2559 scope.go:117] "RemoveContainer" containerID="80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2"
Jul 27 16:58:35.008585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.008567    2559 generic.go:356] "Generic (PLEG): container finished" podID="3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" containerID="45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e" exitCode=0
Jul 27 16:58:35.008696 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.008608    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"
Jul 27 16:58:35.008696 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.008621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g" event={"ID":"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8","Type":"ContainerDied","Data":"45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e"}
Jul 27 16:58:35.008696 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.008660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g" event={"ID":"3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8","Type":"ContainerDied","Data":"0b44a3340acb7a80c325cf13e12ea361b79cac5a42939d42db7eaacdc04671e5"}
Jul 27 16:58:35.015426 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.015410    2559 scope.go:117] "RemoveContainer" containerID="80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2"
Jul 27 16:58:35.015690 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:58:35.015674    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2\": container with ID starting with 80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2 not found: ID does not exist" containerID="80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2"
Jul 27 16:58:35.015725 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.015698    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2"} err="failed to get container status \"80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2\": rpc error: code = NotFound desc = could not find container \"80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2\": container with ID starting with 80e0103726aebf612b91252515bec2cded966738d41f122b6f1f9025c95858f2 not found: ID does not exist"
Jul 27 16:58:35.015725 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.015714    2559 scope.go:117] "RemoveContainer" containerID="45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e"
Jul 27 16:58:35.022894 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.022878    2559 scope.go:117] "RemoveContainer" containerID="45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e"
Jul 27 16:58:35.023138 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:58:35.023118    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e\": container with ID starting with 45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e not found: ID does not exist" containerID="45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e"
Jul 27 16:58:35.023178 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.023145    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e"} err="failed to get container status \"45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e\": rpc error: code = NotFound desc = could not find container \"45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e\": container with ID starting with 45f75e0d78be6a084b39e4b38a1e99150c08bfd54bf89663f8b95941eaaef19e not found: ID does not exist"
Jul 27 16:58:35.035375 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.035354    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"]
Jul 27 16:58:35.042010 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.041991    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6dee51-7d598dc9b-wnh6g"]
Jul 27 16:58:35.056783 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.056756    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"]
Jul 27 16:58:35.060471 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:35.060451    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6dee51-57c7b7d94d-m9dm8"]
Jul 27 16:58:36.735292 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:36.735239    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" path="/var/lib/kubelet/pods/3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8/volumes"
Jul 27 16:58:36.735650 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:36.735640    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0d83af-38c3-4bd1-914d-794a2e9da75c" path="/var/lib/kubelet/pods/7b0d83af-38c3-4bd1-914d-794a2e9da75c/volumes"
Jul 27 16:58:38.553748 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:38.553709    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:38.950715 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:38.950684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:44.799376 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:44.799335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:44.899295 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:44.899243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:45.059964 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:45.059889    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"]
Jul 27 16:58:45.060117 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:45.060069    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy" containerID="cri-o://a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96" gracePeriod=30
Jul 27 16:58:45.079056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:45.079030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:45.326818 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:45.326733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:58:47.867498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:47.867458    2559 patch_prober.go:28] interesting pod/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 16:58:47.867861 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:47.867540    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 16:58:50.309232 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.309202    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:58:50.375615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375527    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-data\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375569    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-token\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375615 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375597    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-credential-socket\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istiod-ca-cert\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375672    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-envoy\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-certs\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375744    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbp5f\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-kube-api-access-cbp5f\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375776    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-podinfo\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.375899 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375836    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-socket\") pod \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\" (UID: \"c68e96da-52bb-4c23-bad5-2fb6f40cdb85\") "
Jul 27 16:58:50.376175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.375961    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:58:50.376175 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.376114    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:58:50.376311 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.376201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:58:50.376311 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.376126    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.376425 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.376384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:58:50.376537 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.376510    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-data" (OuterVolumeSpecName: "istio-data") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:58:50.377717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.377693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-token" (OuterVolumeSpecName: "istio-token") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:50.377859 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.377831    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-kube-api-access-cbp5f" (OuterVolumeSpecName: "kube-api-access-cbp5f") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "kube-api-access-cbp5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:50.378124 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.378099    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:58:50.378224 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.378176    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c68e96da-52bb-4c23-bad5-2fb6f40cdb85" (UID: "c68e96da-52bb-4c23-bad5-2fb6f40cdb85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:58:50.476858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476822    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.476858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476849    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.476858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476858    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.476858 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476866    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbp5f\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-kube-api-access-cbp5f\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.477210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476876    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.477210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476883    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.477210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476891    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:50.477210 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:50.476898    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68e96da-52bb-4c23-bad5-2fb6f40cdb85-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:58:51.082017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.081932    2559 generic.go:356] "Generic (PLEG): container finished" podID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerID="a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96" exitCode=0
Jul 27 16:58:51.082017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.081982    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"
Jul 27 16:58:51.082196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.082013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" event={"ID":"c68e96da-52bb-4c23-bad5-2fb6f40cdb85","Type":"ContainerDied","Data":"a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96"}
Jul 27 16:58:51.082196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.082071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc" event={"ID":"c68e96da-52bb-4c23-bad5-2fb6f40cdb85","Type":"ContainerDied","Data":"369d34b7919f5b9c07247a9184647a5a04cb52d4aab6ab76492a12c62fd72a1a"}
Jul 27 16:58:51.082196 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.082101    2559 scope.go:117] "RemoveContainer" containerID="a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96"
Jul 27 16:58:51.090312 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.090274    2559 scope.go:117] "RemoveContainer" containerID="be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"
Jul 27 16:58:51.097549 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.097534    2559 scope.go:117] "RemoveContainer" containerID="a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96"
Jul 27 16:58:51.097840 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:58:51.097816    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96\": container with ID starting with a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96 not found: ID does not exist" containerID="a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96"
Jul 27 16:58:51.097885 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.097848    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96"} err="failed to get container status \"a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96\": rpc error: code = NotFound desc = could not find container \"a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96\": container with ID starting with a657d94da78641fc0c13d172e2e54e022c767798b69c9ca829f806c11564dd96 not found: ID does not exist"
Jul 27 16:58:51.097885 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.097866    2559 scope.go:117] "RemoveContainer" containerID="be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"
Jul 27 16:58:51.098075 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:58:51.098059    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee\": container with ID starting with be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee not found: ID does not exist" containerID="be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"
Jul 27 16:58:51.098118 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.098082    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee"} err="failed to get container status \"be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee\": rpc error: code = NotFound desc = could not find container \"be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee\": container with ID starting with be330fbd0af95647e70d68cff358983fc38b4f0adadc73b8d707fdf492de00ee not found: ID does not exist"
Jul 27 16:58:51.100613 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.100592    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"]
Jul 27 16:58:51.104820 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:51.104786    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6dee51-openshift-default-5dbd9948c6-kmkqc"]
Jul 27 16:58:52.735341 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:58:52.735308    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" path="/var/lib/kubelet/pods/c68e96da-52bb-4c23-bad5-2fb6f40cdb85/volumes"
Jul 27 16:59:00.932053 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:00.932012    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"]
Jul 27 16:59:00.932533 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:00.932261    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh" podUID="eac93956-3b34-4b0d-8549-067dc25a494a" containerName="maas-api" containerID="cri-o://0852c7e6887ecbc8a1642594f14b84145861ea10cb980c8ea373563a754e50cd" gracePeriod=30
Jul 27 16:59:00.959065 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:00.959040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:01.021528 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.020324    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"]
Jul 27 16:59:01.021528 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.020660    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q" podUID="fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" containerName="payload-pre-processing" containerID="cri-o://12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39" gracePeriod=30
Jul 27 16:59:01.126215 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.126183    2559 generic.go:356] "Generic (PLEG): container finished" podID="eac93956-3b34-4b0d-8549-067dc25a494a" containerID="0852c7e6887ecbc8a1642594f14b84145861ea10cb980c8ea373563a754e50cd" exitCode=0
Jul 27 16:59:01.126397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.126233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh" event={"ID":"eac93956-3b34-4b0d-8549-067dc25a494a","Type":"ContainerDied","Data":"0852c7e6887ecbc8a1642594f14b84145861ea10cb980c8ea373563a754e50cd"}
Jul 27 16:59:01.261366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.261343    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:59:01.297643 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.297620    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:59:01.366373 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366346    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjjbj\" (UniqueName: \"kubernetes.io/projected/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-kube-api-access-rjjbj\") pod \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") "
Jul 27 16:59:01.366533 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366385    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-plugins-config-volume\") pod \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\" (UID: \"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0\") "
Jul 27 16:59:01.366595 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366575    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzrzx\" (UniqueName: \"kubernetes.io/projected/eac93956-3b34-4b0d-8549-067dc25a494a-kube-api-access-xzrzx\") pod \"eac93956-3b34-4b0d-8549-067dc25a494a\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") "
Jul 27 16:59:01.366653 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366642    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac93956-3b34-4b0d-8549-067dc25a494a-maas-api-tls\") pod \"eac93956-3b34-4b0d-8549-067dc25a494a\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") "
Jul 27 16:59:01.366754 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366729    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" (UID: "fefbbd03-1141-4aae-a1ab-43ee67ff1fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:01.366754 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac93956-3b34-4b0d-8549-067dc25a494a-openshift-service-ca\") pod \"eac93956-3b34-4b0d-8549-067dc25a494a\" (UID: \"eac93956-3b34-4b0d-8549-067dc25a494a\") "
Jul 27 16:59:01.366987 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.366969    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:59:01.367065 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.367028    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac93956-3b34-4b0d-8549-067dc25a494a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eac93956-3b34-4b0d-8549-067dc25a494a" (UID: "eac93956-3b34-4b0d-8549-067dc25a494a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:01.368505 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.368483    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac93956-3b34-4b0d-8549-067dc25a494a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eac93956-3b34-4b0d-8549-067dc25a494a" (UID: "eac93956-3b34-4b0d-8549-067dc25a494a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:01.368599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.368557    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-kube-api-access-rjjbj" (OuterVolumeSpecName: "kube-api-access-rjjbj") pod "fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" (UID: "fefbbd03-1141-4aae-a1ab-43ee67ff1fb0"). InnerVolumeSpecName "kube-api-access-rjjbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:59:01.368599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.368579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac93956-3b34-4b0d-8549-067dc25a494a-kube-api-access-xzrzx" (OuterVolumeSpecName: "kube-api-access-xzrzx") pod "eac93956-3b34-4b0d-8549-067dc25a494a" (UID: "eac93956-3b34-4b0d-8549-067dc25a494a"). InnerVolumeSpecName "kube-api-access-xzrzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:59:01.467424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.467349    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzrzx\" (UniqueName: \"kubernetes.io/projected/eac93956-3b34-4b0d-8549-067dc25a494a-kube-api-access-xzrzx\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:59:01.467424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.467377    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac93956-3b34-4b0d-8549-067dc25a494a-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:59:01.467424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.467387    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac93956-3b34-4b0d-8549-067dc25a494a-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:59:01.467424 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:01.467395    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjjbj\" (UniqueName: \"kubernetes.io/projected/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0-kube-api-access-rjjbj\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 16:59:02.131642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.131605    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"
Jul 27 16:59:02.132104 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.131640    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh" event={"ID":"eac93956-3b34-4b0d-8549-067dc25a494a","Type":"ContainerDied","Data":"82b851d5e4a5c81ca167c94b583c42d88748bf7e1bbd0bed44b941552636c05d"}
Jul 27 16:59:02.132104 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.131709    2559 scope.go:117] "RemoveContainer" containerID="0852c7e6887ecbc8a1642594f14b84145861ea10cb980c8ea373563a754e50cd"
Jul 27 16:59:02.132939 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.132885    2559 generic.go:356] "Generic (PLEG): container finished" podID="fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" containerID="12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39" exitCode=0
Jul 27 16:59:02.132939 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.132930    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"
Jul 27 16:59:02.133028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.132941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q" event={"ID":"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0","Type":"ContainerDied","Data":"12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39"}
Jul 27 16:59:02.133028 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.132987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q" event={"ID":"fefbbd03-1141-4aae-a1ab-43ee67ff1fb0","Type":"ContainerDied","Data":"6ba7e0198bfdf03b7d79d3e1aa13804cba34513668c0719b777a8623c5d072c7"}
Jul 27 16:59:02.139829 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.139811    2559 scope.go:117] "RemoveContainer" containerID="12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39"
Jul 27 16:59:02.147940 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.147924    2559 scope.go:117] "RemoveContainer" containerID="12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39"
Jul 27 16:59:02.148211 ip-10-0-140-199 kubenswrapper[2559]: E0727 16:59:02.148193    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39\": container with ID starting with 12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39 not found: ID does not exist" containerID="12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39"
Jul 27 16:59:02.148300 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.148219    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39"} err="failed to get container status \"12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39\": rpc error: code = NotFound desc = could not find container \"12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39\": container with ID starting with 12fb9fe1ed5a7496f2de0b548b8aac78938da6a2d1b81e38ed7b94f688128d39 not found: ID does not exist"
Jul 27 16:59:02.160661 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.160639    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"]
Jul 27 16:59:02.164913 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.164893    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-30f382-548cd5747f-7gxsh"]
Jul 27 16:59:02.176195 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.176170    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"]
Jul 27 16:59:02.181004 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.180982    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-30f382-7b757c7664-5rt6q"]
Jul 27 16:59:02.414397 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.414364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:02.735730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.735649    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac93956-3b34-4b0d-8549-067dc25a494a" path="/var/lib/kubelet/pods/eac93956-3b34-4b0d-8549-067dc25a494a/volumes"
Jul 27 16:59:02.735986 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:02.735974    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" path="/var/lib/kubelet/pods/fefbbd03-1141-4aae-a1ab-43ee67ff1fb0/volumes"
Jul 27 16:59:05.532559 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:05.532527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:06.025479 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:06.025447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:12.091100 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:12.091064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:12.192137 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:12.192096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:12.417265 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:12.417222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:12.644632 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:12.644574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:13.015901 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.015872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:13.398562 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.398532    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"]
Jul 27 16:59:13.399066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399041    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac93956-3b34-4b0d-8549-067dc25a494a" containerName="maas-api"
Jul 27 16:59:13.399066 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399067    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac93956-3b34-4b0d-8549-067dc25a494a" containerName="maas-api"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399089    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0d83af-38c3-4bd1-914d-794a2e9da75c" containerName="maas-api"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399097    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0d83af-38c3-4bd1-914d-794a2e9da75c" containerName="maas-api"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399110    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399119    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399130    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399137    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399151    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" containerName="payload-processing"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399157    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" containerName="payload-processing"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399168    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" containerName="payload-pre-processing"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399173    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" containerName="payload-pre-processing"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399243    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a268b4c-d6ad-4bbc-8991-c5ee4ccb0db8" containerName="payload-processing"
Jul 27 16:59:13.399254 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399254    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0d83af-38c3-4bd1-914d-794a2e9da75c" containerName="maas-api"
Jul 27 16:59:13.399720 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399266    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac93956-3b34-4b0d-8549-067dc25a494a" containerName="maas-api"
Jul 27 16:59:13.399720 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399274    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy"
Jul 27 16:59:13.399720 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399299    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefbbd03-1141-4aae-a1ab-43ee67ff1fb0" containerName="payload-pre-processing"
Jul 27 16:59:13.399720 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.399309    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68e96da-52bb-4c23-bad5-2fb6f40cdb85" containerName="istio-proxy"
Jul 27 16:59:13.404038 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.404016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.407265 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.407246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-f1b699-openshift-default-dockercfg-n7prn\""
Jul 27 16:59:13.416773 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.416748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:13.427929 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.427906    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"]
Jul 27 16:59:13.572064 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-data\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-socket\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9c9\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-kube-api-access-gj9c9\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572226 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-credential-socket\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d769569c-c883-4423-9901-9791c69d55bf-istio-podinfo\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-certs\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d769569c-c883-4423-9901-9791c69d55bf-istiod-ca-cert\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572366 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-istio-token\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.572487 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.572402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-envoy\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.673790 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.673705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-credential-socket\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.673790 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.673740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d769569c-c883-4423-9901-9791c69d55bf-istio-podinfo\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.673790 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.673769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-certs\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.673790 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.673787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d769569c-c883-4423-9901-9791c69d55bf-istiod-ca-cert\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674091 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.673807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-istio-token\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674091 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.673997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-envoy\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674091 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-data\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674246 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-socket\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674246 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9c9\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-kube-api-access-gj9c9\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674246 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-credential-socket\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674432 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-certs\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674483 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674438    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-socket\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674483 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-data\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.674614 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.674592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d769569c-c883-4423-9901-9791c69d55bf-istiod-ca-cert\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.676239 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.676217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-envoy\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.676338 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.676323    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d769569c-c883-4423-9901-9791c69d55bf-istio-podinfo\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.683338 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.683314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-istio-token\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.683561 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.683534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9c9\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-kube-api-access-gj9c9\") pod \"e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") " pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.716332 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.716303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:13.846183 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.846157    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"]
Jul 27 16:59:13.848305 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:59:13.848257    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd769569c_c883_4423_9901_9791c69d55bf.slice/crio-f191723f43dc8c9416c43cd3e56a61d593f2b02957195aa4d47a8a4bfab90fd3 WatchSource:0}: Error finding container f191723f43dc8c9416c43cd3e56a61d593f2b02957195aa4d47a8a4bfab90fd3: Status 404 returned error can't find the container with id f191723f43dc8c9416c43cd3e56a61d593f2b02957195aa4d47a8a4bfab90fd3
Jul 27 16:59:13.850313 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.850264    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:59:13.850383 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.850350    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:59:13.850383 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:13.850379    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:59:14.032468 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:14.032383    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:14.185941 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:14.185904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" event={"ID":"d769569c-c883-4423-9901-9791c69d55bf","Type":"ContainerStarted","Data":"15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"}
Jul 27 16:59:14.186108 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:14.185957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" event={"ID":"d769569c-c883-4423-9901-9791c69d55bf","Type":"ContainerStarted","Data":"f191723f43dc8c9416c43cd3e56a61d593f2b02957195aa4d47a8a4bfab90fd3"}
Jul 27 16:59:14.207509 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:14.207462    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podStartSLOduration=1.207447554 podStartE2EDuration="1.207447554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:14.206239108 +0000 UTC m=+2554.107379556" watchObservedRunningTime="2026-07-27 16:59:14.207447554 +0000 UTC m=+2554.108588000"
Jul 27 16:59:14.716619 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:14.716587    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:14.721192 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:14.721168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:15.190071 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:15.190040    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:15.190997 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:15.190980    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:16.285332 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.285304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:16.391381 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.391335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:16.399076 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.399023    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"]
Jul 27 16:59:16.402462 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.402443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.405124 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.405101    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f1b699-serving-cert\""
Jul 27 16:59:16.405198 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.405122    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8zfn5\""
Jul 27 16:59:16.406447 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.406428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:59:16.406522 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.406429    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:59:16.415540 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.415520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"]
Jul 27 16:59:16.483098 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.483054    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:16.498261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.498232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.498261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.498267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6d5\" (UniqueName: \"kubernetes.io/projected/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-kube-api-access-rh6d5\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.498465 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.498429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.582684 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.582611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:16.599864 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.599834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.600017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.599908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.600017 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.599941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6d5\" (UniqueName: \"kubernetes.io/projected/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-kube-api-access-rh6d5\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.600585 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.600564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.602305 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.602271    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-maas-api-tls\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.608774 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.608754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6d5\" (UniqueName: \"kubernetes.io/projected/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-kube-api-access-rh6d5\") pod \"maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.712255 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.712222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:16.846025 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.843220    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"]
Jul 27 16:59:16.847943 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:59:16.847890    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8006c3_9c72_4fa6_9b56_fa79358a37ae.slice/crio-a27f60c0c3c8b987e4e708942650e7c258bb22ef61625d3d5974d4376838b34c WatchSource:0}: Error finding container a27f60c0c3c8b987e4e708942650e7c258bb22ef61625d3d5974d4376838b34c: Status 404 returned error can't find the container with id a27f60c0c3c8b987e4e708942650e7c258bb22ef61625d3d5974d4376838b34c
Jul 27 16:59:16.973647 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:16.973613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:17.072952 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.072918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:17.092974 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.092930    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"]
Jul 27 16:59:17.097601 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.097539    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.100603 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.100583    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-f1b699\""
Jul 27 16:59:17.100712 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.100682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-f1b699-dockercfg-jx5wk\""
Jul 27 16:59:17.110046 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.110022    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"]
Jul 27 16:59:17.201241 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.201209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8" event={"ID":"fc8006c3-9c72-4fa6-9b56-fa79358a37ae","Type":"ContainerStarted","Data":"a27f60c0c3c8b987e4e708942650e7c258bb22ef61625d3d5974d4376838b34c"}
Jul 27 16:59:17.204884 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.204858    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201bc181-2222-4e7f-a313-23ed9311a795-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.205020 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.204894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnpcp\" (UniqueName: \"kubernetes.io/projected/201bc181-2222-4e7f-a313-23ed9311a795-kube-api-access-jnpcp\") pod \"payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.305450 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.305417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201bc181-2222-4e7f-a313-23ed9311a795-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.305450 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.305456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnpcp\" (UniqueName: \"kubernetes.io/projected/201bc181-2222-4e7f-a313-23ed9311a795-kube-api-access-jnpcp\") pod \"payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.306241 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.306216    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201bc181-2222-4e7f-a313-23ed9311a795-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.316081 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.316058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnpcp\" (UniqueName: \"kubernetes.io/projected/201bc181-2222-4e7f-a313-23ed9311a795-kube-api-access-jnpcp\") pod \"payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.407862 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.407830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:17.537082 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:17.536958    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"]
Jul 27 16:59:17.539573 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:59:17.539537    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201bc181_2222_4e7f_a313_23ed9311a795.slice/crio-d5ba6d12316a624b77e877d942815d46d3796b2804cc3b9240b847052f720a09 WatchSource:0}: Error finding container d5ba6d12316a624b77e877d942815d46d3796b2804cc3b9240b847052f720a09: Status 404 returned error can't find the container with id d5ba6d12316a624b77e877d942815d46d3796b2804cc3b9240b847052f720a09
Jul 27 16:59:18.206642 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.206606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8" event={"ID":"fc8006c3-9c72-4fa6-9b56-fa79358a37ae","Type":"ContainerStarted","Data":"334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864"}
Jul 27 16:59:18.206806 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.206731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:18.208137 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.208109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x" event={"ID":"201bc181-2222-4e7f-a313-23ed9311a795","Type":"ContainerStarted","Data":"f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785"}
Jul 27 16:59:18.208253 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.208157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x" event={"ID":"201bc181-2222-4e7f-a313-23ed9311a795","Type":"ContainerStarted","Data":"d5ba6d12316a624b77e877d942815d46d3796b2804cc3b9240b847052f720a09"}
Jul 27 16:59:18.208253 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.208186    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:18.238484 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.238436    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8" podStartSLOduration=1.893824703 podStartE2EDuration="2.238421593s" totalImagesPullingTime="344.59689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:18.231556664 +0000 UTC m=+2558.132697111" watchObservedRunningTime="2026-07-27 16:59:18.238421593 +0000 UTC m=+2558.139562038"
Jul 27 16:59:18.258832 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:18.258773    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x" podStartSLOduration=1.258759823 podStartE2EDuration="1.258759823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:18.25724187 +0000 UTC m=+2558.158382317" watchObservedRunningTime="2026-07-27 16:59:18.258759823 +0000 UTC m=+2558.159900269"
Jul 27 16:59:20.729029 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:20.728990    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:21.406695 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:21.406648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:21.519925 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:21.519892    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:21.615980 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:21.615938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:24.217365 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.217332    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 16:59:24.808745 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.808694    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:24.887842 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.887804    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"]
Jul 27 16:59:24.888187 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.888163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:24.891040 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.891020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:24.893759 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.893731    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d4529a-serving-cert\""
Jul 27 16:59:24.907995 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.907967    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"]
Jul 27 16:59:24.962675 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:24.962637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:25.074556 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.074469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc0ca6f-003d-4135-ac40-f195bd476e3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.074556 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.074514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7zk\" (UniqueName: \"kubernetes.io/projected/dfc0ca6f-003d-4135-ac40-f195bd476e3b-kube-api-access-4r7zk\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.074556 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.074545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc0ca6f-003d-4135-ac40-f195bd476e3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.175261 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.175221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc0ca6f-003d-4135-ac40-f195bd476e3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.175463 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.175299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7zk\" (UniqueName: \"kubernetes.io/projected/dfc0ca6f-003d-4135-ac40-f195bd476e3b-kube-api-access-4r7zk\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.175463 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.175343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc0ca6f-003d-4135-ac40-f195bd476e3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.176070 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.176051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc0ca6f-003d-4135-ac40-f195bd476e3b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.177765 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.177739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc0ca6f-003d-4135-ac40-f195bd476e3b-maas-api-tls\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.183701 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.183674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7zk\" (UniqueName: \"kubernetes.io/projected/dfc0ca6f-003d-4135-ac40-f195bd476e3b-kube-api-access-4r7zk\") pod \"maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.201649 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.201622    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:25.229191 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.229132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:25.336417 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.336384    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"]
Jul 27 16:59:25.339033 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:59:25.338998    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc0ca6f_003d_4135_ac40_f195bd476e3b.slice/crio-837b17491cac9f411c57d7aeccf4965eec9f0e54aa22326647f1b0b443665be2 WatchSource:0}: Error finding container 837b17491cac9f411c57d7aeccf4965eec9f0e54aa22326647f1b0b443665be2: Status 404 returned error can't find the container with id 837b17491cac9f411c57d7aeccf4965eec9f0e54aa22326647f1b0b443665be2
Jul 27 16:59:25.490703 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.490644    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:25.583803 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.583774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:25.622076 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.621995    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"]
Jul 27 16:59:25.626158 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.626142    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.628790 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.628769    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-d4529a\""
Jul 27 16:59:25.636536 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.636364    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"]
Jul 27 16:59:25.780575 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.780542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.780717 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.780638    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8j4\" (UniqueName: \"kubernetes.io/projected/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-kube-api-access-tl8j4\") pod \"payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.881387 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.881275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.881556 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.881408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8j4\" (UniqueName: \"kubernetes.io/projected/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-kube-api-access-tl8j4\") pod \"payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.882056 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.882035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.890379 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.890357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8j4\" (UniqueName: \"kubernetes.io/projected/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-kube-api-access-tl8j4\") pod \"payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:25.938012 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:25.937969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:26.104861 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.104758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"]
Jul 27 16:59:26.107790 ip-10-0-140-199 kubenswrapper[2559]: W0727 16:59:26.107760    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb8cb50_a076_45fe_8436_e0c6a7ec05c5.slice/crio-1e7ecb0204acc6da9695958d9823a842c1b3f4c0c9659d6999fd66b318679547 WatchSource:0}: Error finding container 1e7ecb0204acc6da9695958d9823a842c1b3f4c0c9659d6999fd66b318679547: Status 404 returned error can't find the container with id 1e7ecb0204acc6da9695958d9823a842c1b3f4c0c9659d6999fd66b318679547
Jul 27 16:59:26.257304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.253054    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c" event={"ID":"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5","Type":"ContainerStarted","Data":"1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1"}
Jul 27 16:59:26.257304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.253122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c" event={"ID":"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5","Type":"ContainerStarted","Data":"1e7ecb0204acc6da9695958d9823a842c1b3f4c0c9659d6999fd66b318679547"}
Jul 27 16:59:26.257304 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.253893    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:26.269837 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.269801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd" event={"ID":"dfc0ca6f-003d-4135-ac40-f195bd476e3b","Type":"ContainerStarted","Data":"2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753"}
Jul 27 16:59:26.270015 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.269901    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:26.270015 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.269919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd" event={"ID":"dfc0ca6f-003d-4135-ac40-f195bd476e3b","Type":"ContainerStarted","Data":"837b17491cac9f411c57d7aeccf4965eec9f0e54aa22326647f1b0b443665be2"}
Jul 27 16:59:26.277297 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.277236    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c" podStartSLOduration=1.2772234550000001 podStartE2EDuration="1.277223455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:26.276685944 +0000 UTC m=+2566.177826393" watchObservedRunningTime="2026-07-27 16:59:26.277223455 +0000 UTC m=+2566.178363902"
Jul 27 16:59:26.297027 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:26.296975    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd" podStartSLOduration=1.945928435 podStartE2EDuration="2.296960971s" totalImagesPullingTime="351.032536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:26.294931493 +0000 UTC m=+2566.196071940" watchObservedRunningTime="2026-07-27 16:59:26.296960971 +0000 UTC m=+2566.198101417"
Jul 27 16:59:29.214149 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:29.214121    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 16:59:29.309140 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:29.309107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:29.782830 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:29.782789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 16:59:31.231464 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:31.231417    2559 patch_prober.go:28] interesting pod/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 16:59:31.231878 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:31.231476    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 16:59:32.279747 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.279678    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 16:59:32.296996 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.296964    2559 generic.go:356] "Generic (PLEG): container finished" podID="d769569c-c883-4423-9901-9791c69d55bf" containerID="15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf" exitCode=0
Jul 27 16:59:32.297162 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.297043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" event={"ID":"d769569c-c883-4423-9901-9791c69d55bf","Type":"ContainerDied","Data":"15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"}
Jul 27 16:59:32.297485 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.297469    2559 scope.go:117] "RemoveContainer" containerID="15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"
Jul 27 16:59:32.298498 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.298470    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:59:32.298599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.298538    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:59:32.298599 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:32.298566    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 16:59:33.303155 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:33.303119    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" event={"ID":"d769569c-c883-4423-9901-9791c69d55bf","Type":"ContainerStarted","Data":"4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0"}
Jul 27 16:59:33.716730 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:33.716697    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:33.716904 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:33.716883    2559 patch_prober.go:28] interesting pod/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 27 16:59:33.716953 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:33.716930    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 27 16:59:34.716972 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:34.716934    2559 patch_prober.go:28] interesting pod/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 27 16:59:34.717404 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:34.716993    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 27 16:59:35.717593 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:35.717558    2559 patch_prober.go:28] interesting pod/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 27 16:59:35.718001 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:35.717608    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 27 16:59:36.717050 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:36.717011    2559 patch_prober.go:28] interesting pod/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 27 16:59:36.717265 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:36.717077    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 27 16:59:37.720259 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:37.720220    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:37.720657 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:37.720607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:37.721266 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:37.721244    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 16:59:38.281257 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:38.281227    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 16:59:42.800947 ip-10-0-140-199 kubenswrapper[2559]: I0727 16:59:42.800913    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:00:13.190617 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:00:13.190583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:00:54.008993 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:00:54.008956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:00:54.109442 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:00:54.109411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:19.015930 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:01:19.015897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:19.119440 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:01:19.119408    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:01:30.280029 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:01:30.279995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:03:14.098770 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:03:14.098732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:03:22.316542 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:03:22.316503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:03:22.417576 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:03:22.417535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:04:39.480436 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:04:39.480356    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:04:41.421922 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:04:41.421890    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:04:48.655615 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:04:48.655571    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:05:28.291762 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:05:28.291721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:06:28.318605 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:06:28.318563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:06:29.194122 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:06:29.194074    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:06:35.949046 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:06:35.949009    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:07:02.662129 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:07:02.662093    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:07:46.433265 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:07:46.433182    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:07:46.529364 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:07:46.529325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:04.780481 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:04.780447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:07.330446 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:07.330407    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:21.764687 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:21.764645    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:25.435911 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:25.435874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:25.781103 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:25.780995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:26.125109 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:26.125006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:28.698194 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.698158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:28.793111 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.793077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:28.914574 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.914537    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"]
Jul 27 17:08:28.919712 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.919686    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:28.922640 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.922620    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2cf641-serving-cert\""
Jul 27 17:08:28.928466 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.928443    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"]
Jul 27 17:08:28.945113 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:28.945082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:29.043659 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.043581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7tw\" (UniqueName: \"kubernetes.io/projected/19ec2be3-167e-42e2-b6ca-011ebb2269fb-kube-api-access-tx7tw\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.043659 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.043628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ec2be3-167e-42e2-b6ca-011ebb2269fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.043835 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.043716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ec2be3-167e-42e2-b6ca-011ebb2269fb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.048077 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.048051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:29.144654 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.144618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ec2be3-167e-42e2-b6ca-011ebb2269fb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.144822 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.144686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7tw\" (UniqueName: \"kubernetes.io/projected/19ec2be3-167e-42e2-b6ca-011ebb2269fb-kube-api-access-tx7tw\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.144822 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.144735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ec2be3-167e-42e2-b6ca-011ebb2269fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.145459 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.145439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ec2be3-167e-42e2-b6ca-011ebb2269fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.147028 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.147009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ec2be3-167e-42e2-b6ca-011ebb2269fb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.154332 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.154304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7tw\" (UniqueName: \"kubernetes.io/projected/19ec2be3-167e-42e2-b6ca-011ebb2269fb-kube-api-access-tx7tw\") pod \"maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.231124 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.231094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:29.367675 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.367649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"]
Jul 27 17:08:29.369566 ip-10-0-140-199 kubenswrapper[2559]: W0727 17:08:29.369534    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ec2be3_167e_42e2_b6ca_011ebb2269fb.slice/crio-3b611541e164f89583df4fb7f3fe568d006a58d1a20194dee75d1d76224f6672 WatchSource:0}: Error finding container 3b611541e164f89583df4fb7f3fe568d006a58d1a20194dee75d1d76224f6672: Status 404 returned error can't find the container with id 3b611541e164f89583df4fb7f3fe568d006a58d1a20194dee75d1d76224f6672
Jul 27 17:08:29.371255 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.371236    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:08:29.439551 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.439517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:29.538113 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.538084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:29.543551 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.543525    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"]
Jul 27 17:08:29.547890 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.547871    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.550777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.550758    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2cf641\""
Jul 27 17:08:29.559534 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.559513    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"]
Jul 27 17:08:29.575905 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.575847    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"]
Jul 27 17:08:29.579326 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.579305    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.583096 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.583081    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2cf641-dockercfg-fkd22\""
Jul 27 17:08:29.588397 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.588374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"]
Jul 27 17:08:29.623769 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.623732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648" event={"ID":"19ec2be3-167e-42e2-b6ca-011ebb2269fb","Type":"ContainerStarted","Data":"3b611541e164f89583df4fb7f3fe568d006a58d1a20194dee75d1d76224f6672"}
Jul 27 17:08:29.649219 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.649192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb97743-545f-4162-a6a7-a7475a8f787f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.649363 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.649230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt9l\" (UniqueName: \"kubernetes.io/projected/314e88b6-bd2d-4c6f-a880-019a6426709a-kube-api-access-dnt9l\") pod \"payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.649363 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.649324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314e88b6-bd2d-4c6f-a880-019a6426709a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.649454 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.649385    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwbt\" (UniqueName: \"kubernetes.io/projected/7bb97743-545f-4162-a6a7-a7475a8f787f-kube-api-access-sbwbt\") pod \"payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.752689 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.752655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314e88b6-bd2d-4c6f-a880-019a6426709a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.752689 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.752698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwbt\" (UniqueName: \"kubernetes.io/projected/7bb97743-545f-4162-a6a7-a7475a8f787f-kube-api-access-sbwbt\") pod \"payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.753217 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.752837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb97743-545f-4162-a6a7-a7475a8f787f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.753217 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.752890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt9l\" (UniqueName: \"kubernetes.io/projected/314e88b6-bd2d-4c6f-a880-019a6426709a-kube-api-access-dnt9l\") pod \"payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.753640 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.753615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314e88b6-bd2d-4c6f-a880-019a6426709a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.753721 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.753652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb97743-545f-4162-a6a7-a7475a8f787f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.765704 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.765677    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt9l\" (UniqueName: \"kubernetes.io/projected/314e88b6-bd2d-4c6f-a880-019a6426709a-kube-api-access-dnt9l\") pod \"payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.765813 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.765702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwbt\" (UniqueName: \"kubernetes.io/projected/7bb97743-545f-4162-a6a7-a7475a8f787f-kube-api-access-sbwbt\") pod \"payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:29.858860 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.858788    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:29.890413 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:29.890392    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:30.043553 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.043520    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"]
Jul 27 17:08:30.046789 ip-10-0-140-199 kubenswrapper[2559]: W0727 17:08:30.046758    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314e88b6_bd2d_4c6f_a880_019a6426709a.slice/crio-e46083cd2174cac89c5893b4bb5a99a4104da42fcc65c240712c30906560f62b WatchSource:0}: Error finding container e46083cd2174cac89c5893b4bb5a99a4104da42fcc65c240712c30906560f62b: Status 404 returned error can't find the container with id e46083cd2174cac89c5893b4bb5a99a4104da42fcc65c240712c30906560f62b
Jul 27 17:08:30.070316 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.070274    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"]
Jul 27 17:08:30.078506 ip-10-0-140-199 kubenswrapper[2559]: W0727 17:08:30.078471    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb97743_545f_4162_a6a7_a7475a8f787f.slice/crio-e7c6f8e222ddcf0012f4d1df30350c909a119dee1c0f1edc674181dfeaeecd77 WatchSource:0}: Error finding container e7c6f8e222ddcf0012f4d1df30350c909a119dee1c0f1edc674181dfeaeecd77: Status 404 returned error can't find the container with id e7c6f8e222ddcf0012f4d1df30350c909a119dee1c0f1edc674181dfeaeecd77
Jul 27 17:08:30.632713 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.632677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x" event={"ID":"7bb97743-545f-4162-a6a7-a7475a8f787f","Type":"ContainerStarted","Data":"b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8"}
Jul 27 17:08:30.632899 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.632733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x" event={"ID":"7bb97743-545f-4162-a6a7-a7475a8f787f","Type":"ContainerStarted","Data":"e7c6f8e222ddcf0012f4d1df30350c909a119dee1c0f1edc674181dfeaeecd77"}
Jul 27 17:08:30.632899 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.632798    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:30.634206 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.634183    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw" event={"ID":"314e88b6-bd2d-4c6f-a880-019a6426709a","Type":"ContainerStarted","Data":"896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e"}
Jul 27 17:08:30.634333 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.634233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw" event={"ID":"314e88b6-bd2d-4c6f-a880-019a6426709a","Type":"ContainerStarted","Data":"e46083cd2174cac89c5893b4bb5a99a4104da42fcc65c240712c30906560f62b"}
Jul 27 17:08:30.634333 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.634264    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:30.635734 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.635710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648" event={"ID":"19ec2be3-167e-42e2-b6ca-011ebb2269fb","Type":"ContainerStarted","Data":"762e76e3edbc131a19f851cc4320292b5886f6cdfb1f9bbea28e25ed37665f79"}
Jul 27 17:08:30.635858 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.635764    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:30.724844 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.724792    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw" podStartSLOduration=1.724774807 podStartE2EDuration="1.724774807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:30.72383783 +0000 UTC m=+3110.624978278" watchObservedRunningTime="2026-07-27 17:08:30.724774807 +0000 UTC m=+3110.625915255"
Jul 27 17:08:30.725410 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.725385    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x" podStartSLOduration=1.7253757809999999 podStartE2EDuration="1.725375781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:30.690141835 +0000 UTC m=+3110.591282283" watchObservedRunningTime="2026-07-27 17:08:30.725375781 +0000 UTC m=+3110.626516277"
Jul 27 17:08:30.764613 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:30.764564    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648" podStartSLOduration=2.260705156 podStartE2EDuration="2.764549521s" totalImagesPullingTime="503.844365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:30.762838737 +0000 UTC m=+3110.663979184" watchObservedRunningTime="2026-07-27 17:08:30.764549521 +0000 UTC m=+3110.665689969"
Jul 27 17:08:33.173620 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:33.173577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:36.646958 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:36.646928    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:08:41.643481 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:41.643447    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:08:41.643900 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:41.643527    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:08:44.106536 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.106499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:44.406549 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.406512    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"]
Jul 27 17:08:44.411176 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.411153    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.413878 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.413860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-de4c74-openshift-default-dockercfg-d9pqj\""
Jul 27 17:08:44.421971 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.421947    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"]
Jul 27 17:08:44.436173 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.436147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:44.489677 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.489643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-credential-socket\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.489677 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.489688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-envoy\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.489920 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.489716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-data\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.489920 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.489840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-socket\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.489920 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.489911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74974\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-kube-api-access-74974\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.490129 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.489948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-certs\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.490129 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.490002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-podinfo\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.490129 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.490062    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff07a830-9374-403f-909d-ce8e79eaee9e-istiod-ca-cert\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.490317 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.490117    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-token\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591084 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-certs\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591084 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-podinfo\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591297 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff07a830-9374-403f-909d-ce8e79eaee9e-istiod-ca-cert\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591297 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-token\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591386 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591331    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-credential-socket\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591386 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591362    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-envoy\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591485 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-data\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591485 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-socket\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591590 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74974\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-kube-api-access-74974\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591590 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591531    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-certs\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591751 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-credential-socket\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591889 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff07a830-9374-403f-909d-ce8e79eaee9e-istiod-ca-cert\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.591977 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-socket\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.592052 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.591996    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-data\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.593575 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.593550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-envoy\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.593675 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.593636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-podinfo\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.599670 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.599643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-token\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.600215 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.600194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74974\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-kube-api-access-74974\") pod \"e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") " pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.721109 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.721030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:44.858308 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.858174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"]
Jul 27 17:08:44.860881 ip-10-0-140-199 kubenswrapper[2559]: W0727 17:08:44.860858    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff07a830_9374_403f_909d_ce8e79eaee9e.slice/crio-ca0f8c451d22c0dac6b1c83bd262e8e72bed11ee265b306d51876fb71da68228 WatchSource:0}: Error finding container ca0f8c451d22c0dac6b1c83bd262e8e72bed11ee265b306d51876fb71da68228: Status 404 returned error can't find the container with id ca0f8c451d22c0dac6b1c83bd262e8e72bed11ee265b306d51876fb71da68228
Jul 27 17:08:44.863368 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.863328    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:08:44.863463 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.863412    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:08:44.863463 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:44.863453    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:08:45.157442 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:45.157401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:45.709585 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:45.709550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" event={"ID":"ff07a830-9374-403f-909d-ce8e79eaee9e","Type":"ContainerStarted","Data":"1be209c58baaefa7d079dfc47000325b5b4ad1f3a374701df84bda24682f5357"}
Jul 27 17:08:45.709760 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:45.709629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" event={"ID":"ff07a830-9374-403f-909d-ce8e79eaee9e","Type":"ContainerStarted","Data":"ca0f8c451d22c0dac6b1c83bd262e8e72bed11ee265b306d51876fb71da68228"}
Jul 27 17:08:45.721666 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:45.721640    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:45.726637 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:45.726610    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:45.734086 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:45.734034    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" podStartSLOduration=1.734012832 podStartE2EDuration="1.734012832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:45.732467577 +0000 UTC m=+3125.633608025" watchObservedRunningTime="2026-07-27 17:08:45.734012832 +0000 UTC m=+3125.635153280"
Jul 27 17:08:46.715468 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:46.715440    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:46.716737 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:46.716710    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:08:47.660959 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:47.660900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:47.758935 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:47.758887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:47.965807 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:47.965696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:48.544026 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:48.543989    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:48.643570 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:48.643531    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:48.725875 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:48.725828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:08:52.273163 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:08:52.273126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:09:02.756472 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:02.756392    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 17:09:02.756867 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:02.756465    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:09:04.798422 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:04.798388    2559 generic.go:356] "Generic (PLEG): container finished" podID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerID="1be209c58baaefa7d079dfc47000325b5b4ad1f3a374701df84bda24682f5357" exitCode=0
Jul 27 17:09:04.798842 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:04.798464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" event={"ID":"ff07a830-9374-403f-909d-ce8e79eaee9e","Type":"ContainerDied","Data":"1be209c58baaefa7d079dfc47000325b5b4ad1f3a374701df84bda24682f5357"}
Jul 27 17:09:04.798887 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:04.798844    2559 scope.go:117] "RemoveContainer" containerID="1be209c58baaefa7d079dfc47000325b5b4ad1f3a374701df84bda24682f5357"
Jul 27 17:09:04.799874 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:04.799850    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:09:04.799934 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:04.799916    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:09:04.799971 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:04.799948    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 27 17:09:05.247761 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:05.247721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:09:05.804571 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:05.804539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" event={"ID":"ff07a830-9374-403f-909d-ce8e79eaee9e","Type":"ContainerStarted","Data":"dfed90fef7e9d4f347873c8ef1a00390ffad539d5fe426da324469f3d2f4ddc4"}
Jul 27 17:09:06.721733 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:06.721701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:09:06.721919 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:06.721834    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 27 17:09:06.721919 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:06.721888    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 27 17:09:07.721462 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:07.721420    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 27 17:09:07.721891 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:07.721477    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 27 17:09:08.721879 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:08.721843    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 27 17:09:08.722268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:08.721912    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 27 17:09:09.358638 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:09.358535    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qxpr8_2e603be2-8915-481b-9db0-4bf942836c39/payload-processing/0.log"
Jul 27 17:09:09.724631 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:09.724602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:09:09.725015 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:09.724850    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:09:09.725465 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:09.725446    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:09:17.499115 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:17.499081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:09:24.875200 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:24.875163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:09:34.791247 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:34.791213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:09:34.847820 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:09:34.847779    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:06.640693 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:06.640663    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x_7bb97743-545f-4162-a6a7-a7475a8f787f/payload-processing/0.log"
Jul 27 17:10:06.764718 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:06.764692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qxpr8_2e603be2-8915-481b-9db0-4bf942836c39/payload-processing/0.log"
Jul 27 17:10:07.182301 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:07.182244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:10.043252 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:10.043215    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:10.146538 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:10.146500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:12.063662 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.063627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:12.374597 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.374502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:12.607252 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.607218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:12.705232 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.705199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:12.796946 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.796912    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"]
Jul 27 17:10:12.799339 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.799318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:12.802409 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.802389    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7787cf\""
Jul 27 17:10:12.802550 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.802502    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7787cf-dockercfg-bd8kd\""
Jul 27 17:10:12.809114 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.809093    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"]
Jul 27 17:10:12.906701 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.906661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8efe61-758f-430d-b383-e247eeb5c9b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:12.906701 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:12.906703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf46\" (UniqueName: \"kubernetes.io/projected/ba8efe61-758f-430d-b383-e247eeb5c9b4-kube-api-access-mjf46\") pod \"payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:13.007745 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:13.007655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8efe61-758f-430d-b383-e247eeb5c9b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:13.007745 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:13.007695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf46\" (UniqueName: \"kubernetes.io/projected/ba8efe61-758f-430d-b383-e247eeb5c9b4-kube-api-access-mjf46\") pod \"payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:13.008441 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:13.008421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8efe61-758f-430d-b383-e247eeb5c9b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:13.016879 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:13.016847    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf46\" (UniqueName: \"kubernetes.io/projected/ba8efe61-758f-430d-b383-e247eeb5c9b4-kube-api-access-mjf46\") pod \"payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:13.110995 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:13.110953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:13.242545 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:13.242481    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"]
Jul 27 17:10:13.245216 ip-10-0-140-199 kubenswrapper[2559]: W0727 17:10:13.245191    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8efe61_758f_430d_b383_e247eeb5c9b4.slice/crio-ff1be2b0db016f2cb835a413735228c86cb910aed415e2f94edff7783b71ceae WatchSource:0}: Error finding container ff1be2b0db016f2cb835a413735228c86cb910aed415e2f94edff7783b71ceae: Status 404 returned error can't find the container with id ff1be2b0db016f2cb835a413735228c86cb910aed415e2f94edff7783b71ceae
Jul 27 17:10:14.118540 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:14.118504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t" event={"ID":"ba8efe61-758f-430d-b383-e247eeb5c9b4","Type":"ContainerStarted","Data":"aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8"}
Jul 27 17:10:14.118932 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:14.118566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t" event={"ID":"ba8efe61-758f-430d-b383-e247eeb5c9b4","Type":"ContainerStarted","Data":"ff1be2b0db016f2cb835a413735228c86cb910aed415e2f94edff7783b71ceae"}
Jul 27 17:10:14.118932 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:14.118640    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:14.136316 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:14.136249    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t" podStartSLOduration=2.13623515 podStartE2EDuration="2.13623515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:14.135447399 +0000 UTC m=+3214.036587848" watchObservedRunningTime="2026-07-27 17:10:14.13623515 +0000 UTC m=+3214.037375611"
Jul 27 17:10:14.946402 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:14.946362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:25.125364 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:25.125331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:43.944406 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:43.944320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:43.969757 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:43.969714    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"]
Jul 27 17:10:43.970025 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:43.969992    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t" podUID="ba8efe61-758f-430d-b383-e247eeb5c9b4" containerName="payload-pre-processing" containerID="cri-o://aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8" gracePeriod=30
Jul 27 17:10:44.069210 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.069172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:44.131030 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.130945    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:44.224497 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.224470    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:44.270986 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.270951    2559 generic.go:356] "Generic (PLEG): container finished" podID="ba8efe61-758f-430d-b383-e247eeb5c9b4" containerID="aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8" exitCode=0
Jul 27 17:10:44.271165 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.270994    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"
Jul 27 17:10:44.271165 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.271028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t" event={"ID":"ba8efe61-758f-430d-b383-e247eeb5c9b4","Type":"ContainerDied","Data":"aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8"}
Jul 27 17:10:44.271165 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.271078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t" event={"ID":"ba8efe61-758f-430d-b383-e247eeb5c9b4","Type":"ContainerDied","Data":"ff1be2b0db016f2cb835a413735228c86cb910aed415e2f94edff7783b71ceae"}
Jul 27 17:10:44.271165 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.271102    2559 scope.go:117] "RemoveContainer" containerID="aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8"
Jul 27 17:10:44.279953 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.279933    2559 scope.go:117] "RemoveContainer" containerID="aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8"
Jul 27 17:10:44.280236 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:10:44.280217    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8\": container with ID starting with aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8 not found: ID does not exist" containerID="aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8"
Jul 27 17:10:44.280329 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.280246    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8"} err="failed to get container status \"aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8\": rpc error: code = NotFound desc = could not find container \"aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8\": container with ID starting with aed7658fee73ed90a21cde56215a8ab7c46acb802e69ce5f7668580ef2aed8e8 not found: ID does not exist"
Jul 27 17:10:44.298584 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.298553    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8efe61-758f-430d-b383-e247eeb5c9b4-plugins-config-volume\") pod \"ba8efe61-758f-430d-b383-e247eeb5c9b4\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") "
Jul 27 17:10:44.298739 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.298661    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjf46\" (UniqueName: \"kubernetes.io/projected/ba8efe61-758f-430d-b383-e247eeb5c9b4-kube-api-access-mjf46\") pod \"ba8efe61-758f-430d-b383-e247eeb5c9b4\" (UID: \"ba8efe61-758f-430d-b383-e247eeb5c9b4\") "
Jul 27 17:10:44.298916 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.298891    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba8efe61-758f-430d-b383-e247eeb5c9b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba8efe61-758f-430d-b383-e247eeb5c9b4" (UID: "ba8efe61-758f-430d-b383-e247eeb5c9b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:10:44.300799 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.300775    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8efe61-758f-430d-b383-e247eeb5c9b4-kube-api-access-mjf46" (OuterVolumeSpecName: "kube-api-access-mjf46") pod "ba8efe61-758f-430d-b383-e247eeb5c9b4" (UID: "ba8efe61-758f-430d-b383-e247eeb5c9b4"). InnerVolumeSpecName "kube-api-access-mjf46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:10:44.399697 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.399663    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjf46\" (UniqueName: \"kubernetes.io/projected/ba8efe61-758f-430d-b383-e247eeb5c9b4-kube-api-access-mjf46\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:10:44.399697 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.399688    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba8efe61-758f-430d-b383-e247eeb5c9b4-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:10:44.593124 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.593087    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"]
Jul 27 17:10:44.598904 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.598878    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7787cf-66b9d587bd-d9s8t"]
Jul 27 17:10:44.736678 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:44.736643    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8efe61-758f-430d-b383-e247eeb5c9b4" path="/var/lib/kubelet/pods/ba8efe61-758f-430d-b383-e247eeb5c9b4/volumes"
Jul 27 17:10:45.656143 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:45.656104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:46.053176 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:46.053086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:52.873350 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:52.873304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:52.973863 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:52.973830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:53.154915 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:53.154865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:53.371374 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:53.371341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:56.357774 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:56.357736    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:10:56.457271 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:10:56.457235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:11:14.259833 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.259799    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"]
Jul 27 17:11:14.260332 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.260112    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648" podUID="19ec2be3-167e-42e2-b6ca-011ebb2269fb" containerName="maas-api" containerID="cri-o://762e76e3edbc131a19f851cc4320292b5886f6cdfb1f9bbea28e25ed37665f79" gracePeriod=30
Jul 27 17:11:14.296442 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.296413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:11:14.319528 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.319500    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"]
Jul 27 17:11:14.319721 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.319694    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x" podUID="7bb97743-545f-4162-a6a7-a7475a8f787f" containerName="payload-processing" containerID="cri-o://b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8" gracePeriod=30
Jul 27 17:11:14.324945 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.324915    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"]
Jul 27 17:11:14.325344 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.325312    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw" podUID="314e88b6-bd2d-4c6f-a880-019a6426709a" containerName="payload-pre-processing" containerID="cri-o://896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e" gracePeriod=30
Jul 27 17:11:14.416182 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.416152    2559 generic.go:356] "Generic (PLEG): container finished" podID="19ec2be3-167e-42e2-b6ca-011ebb2269fb" containerID="762e76e3edbc131a19f851cc4320292b5886f6cdfb1f9bbea28e25ed37665f79" exitCode=0
Jul 27 17:11:14.416369 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.416205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648" event={"ID":"19ec2be3-167e-42e2-b6ca-011ebb2269fb","Type":"ContainerDied","Data":"762e76e3edbc131a19f851cc4320292b5886f6cdfb1f9bbea28e25ed37665f79"}
Jul 27 17:11:14.476669 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.476494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:11:14.612444 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.612415    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:11:14.617360 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.617339    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:11:14.628824 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.628805    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:11:14.659610 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659580    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ec2be3-167e-42e2-b6ca-011ebb2269fb-openshift-service-ca\") pod \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") "
Jul 27 17:11:14.659791 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnt9l\" (UniqueName: \"kubernetes.io/projected/314e88b6-bd2d-4c6f-a880-019a6426709a-kube-api-access-dnt9l\") pod \"314e88b6-bd2d-4c6f-a880-019a6426709a\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") "
Jul 27 17:11:14.659791 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659656    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb97743-545f-4162-a6a7-a7475a8f787f-plugins-config-volume\") pod \"7bb97743-545f-4162-a6a7-a7475a8f787f\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") "
Jul 27 17:11:14.659791 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659705    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314e88b6-bd2d-4c6f-a880-019a6426709a-plugins-config-volume\") pod \"314e88b6-bd2d-4c6f-a880-019a6426709a\" (UID: \"314e88b6-bd2d-4c6f-a880-019a6426709a\") "
Jul 27 17:11:14.659791 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659773    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx7tw\" (UniqueName: \"kubernetes.io/projected/19ec2be3-167e-42e2-b6ca-011ebb2269fb-kube-api-access-tx7tw\") pod \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") "
Jul 27 17:11:14.660028 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ec2be3-167e-42e2-b6ca-011ebb2269fb-maas-api-tls\") pod \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\" (UID: \"19ec2be3-167e-42e2-b6ca-011ebb2269fb\") "
Jul 27 17:11:14.660028 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.659859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbwbt\" (UniqueName: \"kubernetes.io/projected/7bb97743-545f-4162-a6a7-a7475a8f787f-kube-api-access-sbwbt\") pod \"7bb97743-545f-4162-a6a7-a7475a8f787f\" (UID: \"7bb97743-545f-4162-a6a7-a7475a8f787f\") "
Jul 27 17:11:14.660180 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.660077    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb97743-545f-4162-a6a7-a7475a8f787f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bb97743-545f-4162-a6a7-a7475a8f787f" (UID: "7bb97743-545f-4162-a6a7-a7475a8f787f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:11:14.660469 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.660418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ec2be3-167e-42e2-b6ca-011ebb2269fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "19ec2be3-167e-42e2-b6ca-011ebb2269fb" (UID: "19ec2be3-167e-42e2-b6ca-011ebb2269fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:11:14.660592 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.660547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314e88b6-bd2d-4c6f-a880-019a6426709a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "314e88b6-bd2d-4c6f-a880-019a6426709a" (UID: "314e88b6-bd2d-4c6f-a880-019a6426709a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:11:14.662257 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.662225    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb97743-545f-4162-a6a7-a7475a8f787f-kube-api-access-sbwbt" (OuterVolumeSpecName: "kube-api-access-sbwbt") pod "7bb97743-545f-4162-a6a7-a7475a8f787f" (UID: "7bb97743-545f-4162-a6a7-a7475a8f787f"). InnerVolumeSpecName "kube-api-access-sbwbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:11:14.662463 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.662439    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ec2be3-167e-42e2-b6ca-011ebb2269fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "19ec2be3-167e-42e2-b6ca-011ebb2269fb" (UID: "19ec2be3-167e-42e2-b6ca-011ebb2269fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:11:14.662514 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.662449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314e88b6-bd2d-4c6f-a880-019a6426709a-kube-api-access-dnt9l" (OuterVolumeSpecName: "kube-api-access-dnt9l") pod "314e88b6-bd2d-4c6f-a880-019a6426709a" (UID: "314e88b6-bd2d-4c6f-a880-019a6426709a"). InnerVolumeSpecName "kube-api-access-dnt9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:11:14.662639 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.662615    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ec2be3-167e-42e2-b6ca-011ebb2269fb-kube-api-access-tx7tw" (OuterVolumeSpecName: "kube-api-access-tx7tw") pod "19ec2be3-167e-42e2-b6ca-011ebb2269fb" (UID: "19ec2be3-167e-42e2-b6ca-011ebb2269fb"). InnerVolumeSpecName "kube-api-access-tx7tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:11:14.761132 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761106    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314e88b6-bd2d-4c6f-a880-019a6426709a-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:14.761132 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761132    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7tw\" (UniqueName: \"kubernetes.io/projected/19ec2be3-167e-42e2-b6ca-011ebb2269fb-kube-api-access-tx7tw\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:14.761264 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761142    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19ec2be3-167e-42e2-b6ca-011ebb2269fb-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:14.761264 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761150    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbwbt\" (UniqueName: \"kubernetes.io/projected/7bb97743-545f-4162-a6a7-a7475a8f787f-kube-api-access-sbwbt\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:14.761264 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761159    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19ec2be3-167e-42e2-b6ca-011ebb2269fb-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:14.761264 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761168    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnt9l\" (UniqueName: \"kubernetes.io/projected/314e88b6-bd2d-4c6f-a880-019a6426709a-kube-api-access-dnt9l\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:14.761264 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:14.761177    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb97743-545f-4162-a6a7-a7475a8f787f-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:11:15.422457 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.422421    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bb97743-545f-4162-a6a7-a7475a8f787f" containerID="b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8" exitCode=0
Jul 27 17:11:15.422957 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.422467    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"
Jul 27 17:11:15.422957 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.422499    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x" event={"ID":"7bb97743-545f-4162-a6a7-a7475a8f787f","Type":"ContainerDied","Data":"b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8"}
Jul 27 17:11:15.422957 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.422553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x" event={"ID":"7bb97743-545f-4162-a6a7-a7475a8f787f","Type":"ContainerDied","Data":"e7c6f8e222ddcf0012f4d1df30350c909a119dee1c0f1edc674181dfeaeecd77"}
Jul 27 17:11:15.422957 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.422584    2559 scope.go:117] "RemoveContainer" containerID="b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8"
Jul 27 17:11:15.423914 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.423783    2559 generic.go:356] "Generic (PLEG): container finished" podID="314e88b6-bd2d-4c6f-a880-019a6426709a" containerID="896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e" exitCode=0
Jul 27 17:11:15.423914 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.423841    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"
Jul 27 17:11:15.423914 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.423894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw" event={"ID":"314e88b6-bd2d-4c6f-a880-019a6426709a","Type":"ContainerDied","Data":"896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e"}
Jul 27 17:11:15.424107 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.423944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw" event={"ID":"314e88b6-bd2d-4c6f-a880-019a6426709a","Type":"ContainerDied","Data":"e46083cd2174cac89c5893b4bb5a99a4104da42fcc65c240712c30906560f62b"}
Jul 27 17:11:15.425516 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.425492    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648" event={"ID":"19ec2be3-167e-42e2-b6ca-011ebb2269fb","Type":"ContainerDied","Data":"3b611541e164f89583df4fb7f3fe568d006a58d1a20194dee75d1d76224f6672"}
Jul 27 17:11:15.425516 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.425510    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"
Jul 27 17:11:15.431982 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.431962    2559 scope.go:117] "RemoveContainer" containerID="b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8"
Jul 27 17:11:15.432223 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:11:15.432203    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8\": container with ID starting with b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8 not found: ID does not exist" containerID="b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8"
Jul 27 17:11:15.432265 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.432231    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8"} err="failed to get container status \"b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8\": rpc error: code = NotFound desc = could not find container \"b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8\": container with ID starting with b6d03c2add62616ccc037e0dff54971a9a7640abe36689b626d0921675a253c8 not found: ID does not exist"
Jul 27 17:11:15.432265 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.432248    2559 scope.go:117] "RemoveContainer" containerID="896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e"
Jul 27 17:11:15.441593 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.441571    2559 scope.go:117] "RemoveContainer" containerID="896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e"
Jul 27 17:11:15.441879 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:11:15.441860    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e\": container with ID starting with 896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e not found: ID does not exist" containerID="896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e"
Jul 27 17:11:15.441950 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.441887    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e"} err="failed to get container status \"896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e\": rpc error: code = NotFound desc = could not find container \"896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e\": container with ID starting with 896e40f333e9acbef34673b10050b455021f636f3b1270a27e4d83e3bed9c65e not found: ID does not exist"
Jul 27 17:11:15.441950 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.441903    2559 scope.go:117] "RemoveContainer" containerID="762e76e3edbc131a19f851cc4320292b5886f6cdfb1f9bbea28e25ed37665f79"
Jul 27 17:11:15.452775 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.452754    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"]
Jul 27 17:11:15.459192 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.459157    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cf641-648b4f7966-zh56x"]
Jul 27 17:11:15.469736 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.469714    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"]
Jul 27 17:11:15.473765 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.473741    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cf641-dbc755d6d-6ptdw"]
Jul 27 17:11:15.487326 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.487295    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"]
Jul 27 17:11:15.490550 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:15.490530    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cf641-58f44b87f5-6p648"]
Jul 27 17:11:16.736273 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:16.736240    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19ec2be3-167e-42e2-b6ca-011ebb2269fb" path="/var/lib/kubelet/pods/19ec2be3-167e-42e2-b6ca-011ebb2269fb/volumes"
Jul 27 17:11:16.736699 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:16.736652    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="314e88b6-bd2d-4c6f-a880-019a6426709a" path="/var/lib/kubelet/pods/314e88b6-bd2d-4c6f-a880-019a6426709a/volumes"
Jul 27 17:11:16.736966 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:16.736952    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bb97743-545f-4162-a6a7-a7475a8f787f" path="/var/lib/kubelet/pods/7bb97743-545f-4162-a6a7-a7475a8f787f/volumes"
Jul 27 17:11:16.927255 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:16.927221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:11:17.349979 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:17.349938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:11:22.404233 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:22.404184    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:11:58.754200 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:11:58.754143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:00.301355 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:00.301325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:06.550557 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:06.550522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:06.619048 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:06.619011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:06.855179 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:06.855097    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:07.096085 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:07.096047    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:17.664270 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:17.664229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:17.742800 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:17.742768    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:17.842218 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:17.842187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:22.282720 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:22.282678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:22.693212 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:22.693177    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:29.336209 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:29.336167    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:29.436975 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:29.436934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:29.591328 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:29.591236    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"]
Jul 27 17:12:29.591491 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:29.591466    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy" containerID="cri-o://dfed90fef7e9d4f347873c8ef1a00390ffad539d5fe426da324469f3d2f4ddc4" gracePeriod=30
Jul 27 17:12:29.615984 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:29.615954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:29.853612 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:29.853517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:34.795736 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.795698    2559 generic.go:356] "Generic (PLEG): container finished" podID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerID="dfed90fef7e9d4f347873c8ef1a00390ffad539d5fe426da324469f3d2f4ddc4" exitCode=0
Jul 27 17:12:34.796169 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.795778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" event={"ID":"ff07a830-9374-403f-909d-ce8e79eaee9e","Type":"ContainerDied","Data":"dfed90fef7e9d4f347873c8ef1a00390ffad539d5fe426da324469f3d2f4ddc4"}
Jul 27 17:12:34.796169 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.795857    2559 scope.go:117] "RemoveContainer" containerID="1be209c58baaefa7d079dfc47000325b5b4ad1f3a374701df84bda24682f5357"
Jul 27 17:12:34.855162 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.855107    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:12:34.976002 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.975967    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-certs\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976197 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976009    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-data\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976197 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976054    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-podinfo\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976197 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976084    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-socket\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976197 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-token\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976197 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976137    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-credential-socket\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976197 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff07a830-9374-403f-909d-ce8e79eaee9e-istiod-ca-cert\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976534 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976224    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74974\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-kube-api-access-74974\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976534 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976263    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-envoy\") pod \"ff07a830-9374-403f-909d-ce8e79eaee9e\" (UID: \"ff07a830-9374-403f-909d-ce8e79eaee9e\") "
Jul 27 17:12:34.976534 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976308    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:34.976534 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976423    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:34.976743 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976597    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:34.976743 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976595    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:34.976743 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976616    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:34.976743 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976600    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff07a830-9374-403f-909d-ce8e79eaee9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:12:34.976949 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.976865    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:34.978554 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.978523    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:34.978689 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.978552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-kube-api-access-74974" (OuterVolumeSpecName: "kube-api-access-74974") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "kube-api-access-74974". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:34.978689 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.978614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:12:34.978689 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:34.978620    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff07a830-9374-403f-909d-ce8e79eaee9e" (UID: "ff07a830-9374-403f-909d-ce8e79eaee9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:35.077634 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077600    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.077634 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077626    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.077634 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077636    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff07a830-9374-403f-909d-ce8e79eaee9e-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.077634 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077645    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74974\" (UniqueName: \"kubernetes.io/projected/ff07a830-9374-403f-909d-ce8e79eaee9e-kube-api-access-74974\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.077905 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077655    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.077905 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077665    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.077905 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.077673    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff07a830-9374-403f-909d-ce8e79eaee9e-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:35.802046 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.802019    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"
Jul 27 17:12:35.802471 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.802016    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v" event={"ID":"ff07a830-9374-403f-909d-ce8e79eaee9e","Type":"ContainerDied","Data":"ca0f8c451d22c0dac6b1c83bd262e8e72bed11ee265b306d51876fb71da68228"}
Jul 27 17:12:35.802471 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.802165    2559 scope.go:117] "RemoveContainer" containerID="dfed90fef7e9d4f347873c8ef1a00390ffad539d5fe426da324469f3d2f4ddc4"
Jul 27 17:12:35.826673 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.826649    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"]
Jul 27 17:12:35.830331 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:35.830306    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-de4c74-openshift-default-854d957786-vtn9v"]
Jul 27 17:12:36.736215 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:36.736180    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" path="/var/lib/kubelet/pods/ff07a830-9374-403f-909d-ce8e79eaee9e/volumes"
Jul 27 17:12:41.450401 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.450366    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"]
Jul 27 17:12:41.451175 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.451120    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8" podUID="fc8006c3-9c72-4fa6-9b56-fa79358a37ae" containerName="maas-api" containerID="cri-o://334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864" gracePeriod=30
Jul 27 17:12:41.484962 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.484930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:41.512604 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.512567    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"]
Jul 27 17:12:41.512827 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.512796    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x" podUID="201bc181-2222-4e7f-a313-23ed9311a795" containerName="payload-pre-processing" containerID="cri-o://f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785" gracePeriod=30
Jul 27 17:12:41.547111 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.547075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:41.650470 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.650431    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:41.793691 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.793665    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 17:12:41.796503 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.796477    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 17:12:41.833807 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.833708    2559 generic.go:356] "Generic (PLEG): container finished" podID="fc8006c3-9c72-4fa6-9b56-fa79358a37ae" containerID="334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864" exitCode=0
Jul 27 17:12:41.833807 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.833768    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"
Jul 27 17:12:41.833807 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.833772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8" event={"ID":"fc8006c3-9c72-4fa6-9b56-fa79358a37ae","Type":"ContainerDied","Data":"334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864"}
Jul 27 17:12:41.834118 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.833874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8" event={"ID":"fc8006c3-9c72-4fa6-9b56-fa79358a37ae","Type":"ContainerDied","Data":"a27f60c0c3c8b987e4e708942650e7c258bb22ef61625d3d5974d4376838b34c"}
Jul 27 17:12:41.834118 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.833924    2559 scope.go:117] "RemoveContainer" containerID="334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864"
Jul 27 17:12:41.835506 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.835482    2559 generic.go:356] "Generic (PLEG): container finished" podID="201bc181-2222-4e7f-a313-23ed9311a795" containerID="f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785" exitCode=0
Jul 27 17:12:41.835618 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.835508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x" event={"ID":"201bc181-2222-4e7f-a313-23ed9311a795","Type":"ContainerDied","Data":"f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785"}
Jul 27 17:12:41.835618 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.835579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x" event={"ID":"201bc181-2222-4e7f-a313-23ed9311a795","Type":"ContainerDied","Data":"d5ba6d12316a624b77e877d942815d46d3796b2804cc3b9240b847052f720a09"}
Jul 27 17:12:41.835618 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.835534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"
Jul 27 17:12:41.843564 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.843538    2559 scope.go:117] "RemoveContainer" containerID="334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864"
Jul 27 17:12:41.843854 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:12:41.843835    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864\": container with ID starting with 334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864 not found: ID does not exist" containerID="334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864"
Jul 27 17:12:41.843946 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.843867    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864"} err="failed to get container status \"334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864\": rpc error: code = NotFound desc = could not find container \"334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864\": container with ID starting with 334ee68958db48cf24a2ffa5be0f9ddb3a5e1b6d2d87da7b3760b2709068e864 not found: ID does not exist"
Jul 27 17:12:41.843946 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.843890    2559 scope.go:117] "RemoveContainer" containerID="f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785"
Jul 27 17:12:41.852342 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.852319    2559 scope.go:117] "RemoveContainer" containerID="f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785"
Jul 27 17:12:41.852657 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:12:41.852630    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785\": container with ID starting with f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785 not found: ID does not exist" containerID="f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785"
Jul 27 17:12:41.852733 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.852668    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785"} err="failed to get container status \"f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785\": rpc error: code = NotFound desc = could not find container \"f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785\": container with ID starting with f4be3fc20be111d904a06a43982443b3eeca4e6bdf0eba9d829b11fff8670785 not found: ID does not exist"
Jul 27 17:12:41.939157 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939118    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201bc181-2222-4e7f-a313-23ed9311a795-plugins-config-volume\") pod \"201bc181-2222-4e7f-a313-23ed9311a795\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") "
Jul 27 17:12:41.939357 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939171    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-openshift-service-ca\") pod \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") "
Jul 27 17:12:41.939357 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh6d5\" (UniqueName: \"kubernetes.io/projected/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-kube-api-access-rh6d5\") pod \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") "
Jul 27 17:12:41.939357 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939213    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnpcp\" (UniqueName: \"kubernetes.io/projected/201bc181-2222-4e7f-a313-23ed9311a795-kube-api-access-jnpcp\") pod \"201bc181-2222-4e7f-a313-23ed9311a795\" (UID: \"201bc181-2222-4e7f-a313-23ed9311a795\") "
Jul 27 17:12:41.939357 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939317    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-maas-api-tls\") pod \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\" (UID: \"fc8006c3-9c72-4fa6-9b56-fa79358a37ae\") "
Jul 27 17:12:41.939721 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939567    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201bc181-2222-4e7f-a313-23ed9311a795-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "201bc181-2222-4e7f-a313-23ed9311a795" (UID: "201bc181-2222-4e7f-a313-23ed9311a795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:12:41.939783 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.939726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc8006c3-9c72-4fa6-9b56-fa79358a37ae" (UID: "fc8006c3-9c72-4fa6-9b56-fa79358a37ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:12:41.941516 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.941483    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc8006c3-9c72-4fa6-9b56-fa79358a37ae" (UID: "fc8006c3-9c72-4fa6-9b56-fa79358a37ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:12:41.941516 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.941513    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-kube-api-access-rh6d5" (OuterVolumeSpecName: "kube-api-access-rh6d5") pod "fc8006c3-9c72-4fa6-9b56-fa79358a37ae" (UID: "fc8006c3-9c72-4fa6-9b56-fa79358a37ae"). InnerVolumeSpecName "kube-api-access-rh6d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:41.941665 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:41.941499    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201bc181-2222-4e7f-a313-23ed9311a795-kube-api-access-jnpcp" (OuterVolumeSpecName: "kube-api-access-jnpcp") pod "201bc181-2222-4e7f-a313-23ed9311a795" (UID: "201bc181-2222-4e7f-a313-23ed9311a795"). InnerVolumeSpecName "kube-api-access-jnpcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:42.040793 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.040707    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:42.040793 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.040738    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/201bc181-2222-4e7f-a313-23ed9311a795-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:42.040793 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.040750    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:42.040793 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.040759    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6d5\" (UniqueName: \"kubernetes.io/projected/fc8006c3-9c72-4fa6-9b56-fa79358a37ae-kube-api-access-rh6d5\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:42.040793 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.040767    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnpcp\" (UniqueName: \"kubernetes.io/projected/201bc181-2222-4e7f-a313-23ed9311a795-kube-api-access-jnpcp\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:42.159832 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.159796    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"]
Jul 27 17:12:42.166544 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.166516    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f1b699-6f9bfb6696-fr4r8"]
Jul 27 17:12:42.180133 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.180108    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"]
Jul 27 17:12:42.182391 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.182368    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-f1b699-85d99ff56d-tj55x"]
Jul 27 17:12:42.735590 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.735513    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201bc181-2222-4e7f-a313-23ed9311a795" path="/var/lib/kubelet/pods/201bc181-2222-4e7f-a313-23ed9311a795/volumes"
Jul 27 17:12:42.735931 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:42.735883    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8006c3-9c72-4fa6-9b56-fa79358a37ae" path="/var/lib/kubelet/pods/fc8006c3-9c72-4fa6-9b56-fa79358a37ae/volumes"
Jul 27 17:12:45.093479 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:45.093438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:45.464557 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:45.464527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:51.717649 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:51.717609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:51.780590 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:51.780554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:51.963705 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:51.963676    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"]
Jul 27 17:12:51.963964 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:51.963937    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" containerID="cri-o://4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0" gracePeriod=30
Jul 27 17:12:51.996983 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:51.996883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:52.228616 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:52.228576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:12:52.723457 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:52.723417    2559 patch_prober.go:28] interesting pod/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 17:12:52.723842 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:52.723518    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:12:57.219702 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.219679    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 17:12:57.256885 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.256805    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-socket\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257024 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.256943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj9c9\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-kube-api-access-gj9c9\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257024 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.256980    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d769569c-c883-4423-9901-9791c69d55bf-istio-podinfo\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257139 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257031    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-certs\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257139 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257062    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-data\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257139 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257092    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d769569c-c883-4423-9901-9791c69d55bf-istiod-ca-cert\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257317 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-istio-token\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257317 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257230    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-envoy\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257317 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257260    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:57.257317 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257299    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-credential-socket\") pod \"d769569c-c883-4423-9901-9791c69d55bf\" (UID: \"d769569c-c883-4423-9901-9791c69d55bf\") "
Jul 27 17:12:57.257533 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257433    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:57.257855 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d769569c-c883-4423-9901-9791c69d55bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:12:57.257855 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257685    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:57.257855 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257679    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-certs\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.257855 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.257735    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-workload-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.259058 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.259024    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:57.259699 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.259669    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-kube-api-access-gj9c9" (OuterVolumeSpecName: "kube-api-access-gj9c9") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "kube-api-access-gj9c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:57.259877 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.259859    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:57.259999 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.259983    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d769569c-c883-4423-9901-9791c69d55bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:12:57.260139 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.260125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d769569c-c883-4423-9901-9791c69d55bf" (UID: "d769569c-c883-4423-9901-9791c69d55bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:57.358752 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358717    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-data\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.358752 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358753    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d769569c-c883-4423-9901-9791c69d55bf-istiod-ca-cert\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.358752 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358767    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-istio-token\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.359053 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358780    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-istio-envoy\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.359053 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358792    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d769569c-c883-4423-9901-9791c69d55bf-credential-socket\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.359053 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358805    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9c9\" (UniqueName: \"kubernetes.io/projected/d769569c-c883-4423-9901-9791c69d55bf-kube-api-access-gj9c9\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.359053 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.358819    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d769569c-c883-4423-9901-9791c69d55bf-istio-podinfo\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:12:57.913556 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.913521    2559 generic.go:356] "Generic (PLEG): container finished" podID="d769569c-c883-4423-9901-9791c69d55bf" containerID="4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0" exitCode=0
Jul 27 17:12:57.913742 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.913570    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"
Jul 27 17:12:57.913742 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.913589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" event={"ID":"d769569c-c883-4423-9901-9791c69d55bf","Type":"ContainerDied","Data":"4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0"}
Jul 27 17:12:57.913742 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.913633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz" event={"ID":"d769569c-c883-4423-9901-9791c69d55bf","Type":"ContainerDied","Data":"f191723f43dc8c9416c43cd3e56a61d593f2b02957195aa4d47a8a4bfab90fd3"}
Jul 27 17:12:57.913742 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.913654    2559 scope.go:117] "RemoveContainer" containerID="4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0"
Jul 27 17:12:57.924030 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.924012    2559 scope.go:117] "RemoveContainer" containerID="15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"
Jul 27 17:12:57.931275 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.931256    2559 scope.go:117] "RemoveContainer" containerID="4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0"
Jul 27 17:12:57.931637 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:12:57.931611    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0\": container with ID starting with 4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0 not found: ID does not exist" containerID="4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0"
Jul 27 17:12:57.931706 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.931639    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0"} err="failed to get container status \"4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0\": rpc error: code = NotFound desc = could not find container \"4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0\": container with ID starting with 4516f54915fd68a81bec57b1051f7f3ac37d7da9cdfe871b574c3dae7fa839e0 not found: ID does not exist"
Jul 27 17:12:57.931706 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.931656    2559 scope.go:117] "RemoveContainer" containerID="15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"
Jul 27 17:12:57.931888 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:12:57.931871    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf\": container with ID starting with 15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf not found: ID does not exist" containerID="15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"
Jul 27 17:12:57.931933 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.931892    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf"} err="failed to get container status \"15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf\": rpc error: code = NotFound desc = could not find container \"15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf\": container with ID starting with 15e7793a5c87d4fd0a817ba3292abda46fa7cc14efe879b2e1af32f09051acaf not found: ID does not exist"
Jul 27 17:12:57.937878 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.937856    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"]
Jul 27 17:12:57.939378 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:57.939353    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-f1b699-openshift-default-8698b8bb8-krnkz"]
Jul 27 17:12:58.736430 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:12:58.736391    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d769569c-c883-4423-9901-9791c69d55bf" path="/var/lib/kubelet/pods/d769569c-c883-4423-9901-9791c69d55bf/volumes"
Jul 27 17:13:03.058194 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.058154    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"]
Jul 27 17:13:03.058685 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.058497    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd" podUID="dfc0ca6f-003d-4135-ac40-f195bd476e3b" containerName="maas-api" containerID="cri-o://2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753" gracePeriod=30
Jul 27 17:13:03.127648 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.127613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:03.227694 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.227657    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"]
Jul 27 17:13:03.227940 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.227913    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c" podUID="4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" containerName="payload-processing" containerID="cri-o://1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1" gracePeriod=30
Jul 27 17:13:03.276624 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.276580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:03.351939 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.351916    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 17:13:03.368755 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.368720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:03.409722 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.409690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r7zk\" (UniqueName: \"kubernetes.io/projected/dfc0ca6f-003d-4135-ac40-f195bd476e3b-kube-api-access-4r7zk\") pod \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") "
Jul 27 17:13:03.409950 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.409820    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc0ca6f-003d-4135-ac40-f195bd476e3b-maas-api-tls\") pod \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") "
Jul 27 17:13:03.409950 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.409856    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc0ca6f-003d-4135-ac40-f195bd476e3b-openshift-service-ca\") pod \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\" (UID: \"dfc0ca6f-003d-4135-ac40-f195bd476e3b\") "
Jul 27 17:13:03.410419 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.410392    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc0ca6f-003d-4135-ac40-f195bd476e3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfc0ca6f-003d-4135-ac40-f195bd476e3b" (UID: "dfc0ca6f-003d-4135-ac40-f195bd476e3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:13:03.412965 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.412937    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc0ca6f-003d-4135-ac40-f195bd476e3b-kube-api-access-4r7zk" (OuterVolumeSpecName: "kube-api-access-4r7zk") pod "dfc0ca6f-003d-4135-ac40-f195bd476e3b" (UID: "dfc0ca6f-003d-4135-ac40-f195bd476e3b"). InnerVolumeSpecName "kube-api-access-4r7zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:13:03.413991 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.413967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc0ca6f-003d-4135-ac40-f195bd476e3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfc0ca6f-003d-4135-ac40-f195bd476e3b" (UID: "dfc0ca6f-003d-4135-ac40-f195bd476e3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:13:03.482712 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.482686    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 17:13:03.510262 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.510235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-plugins-config-volume\") pod \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") "
Jul 27 17:13:03.510444 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.510338    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl8j4\" (UniqueName: \"kubernetes.io/projected/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-kube-api-access-tl8j4\") pod \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\" (UID: \"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5\") "
Jul 27 17:13:03.510611 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.510592    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfc0ca6f-003d-4135-ac40-f195bd476e3b-maas-api-tls\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:13:03.510690 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.510618    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfc0ca6f-003d-4135-ac40-f195bd476e3b-openshift-service-ca\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:13:03.510690 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.510635    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7zk\" (UniqueName: \"kubernetes.io/projected/dfc0ca6f-003d-4135-ac40-f195bd476e3b-kube-api-access-4r7zk\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:13:03.510690 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.510640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" (UID: "4cb8cb50-a076-45fe-8436-e0c6a7ec05c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:13:03.512329 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.512305    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-kube-api-access-tl8j4" (OuterVolumeSpecName: "kube-api-access-tl8j4") pod "4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" (UID: "4cb8cb50-a076-45fe-8436-e0c6a7ec05c5"). InnerVolumeSpecName "kube-api-access-tl8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:13:03.611484 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.611399    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-plugins-config-volume\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:13:03.611484 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.611428    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8j4\" (UniqueName: \"kubernetes.io/projected/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5-kube-api-access-tl8j4\") on node \"ip-10-0-140-199.ec2.internal\" DevicePath \"\""
Jul 27 17:13:03.942056 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.942019    2559 generic.go:356] "Generic (PLEG): container finished" podID="4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" containerID="1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1" exitCode=0
Jul 27 17:13:03.942260 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.942070    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"
Jul 27 17:13:03.942260 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.942094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c" event={"ID":"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5","Type":"ContainerDied","Data":"1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1"}
Jul 27 17:13:03.942260 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.942138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c" event={"ID":"4cb8cb50-a076-45fe-8436-e0c6a7ec05c5","Type":"ContainerDied","Data":"1e7ecb0204acc6da9695958d9823a842c1b3f4c0c9659d6999fd66b318679547"}
Jul 27 17:13:03.942260 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.942168    2559 scope.go:117] "RemoveContainer" containerID="1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1"
Jul 27 17:13:03.943387 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.943363    2559 generic.go:356] "Generic (PLEG): container finished" podID="dfc0ca6f-003d-4135-ac40-f195bd476e3b" containerID="2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753" exitCode=0
Jul 27 17:13:03.943462 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.943423    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"
Jul 27 17:13:03.943516 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.943451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd" event={"ID":"dfc0ca6f-003d-4135-ac40-f195bd476e3b","Type":"ContainerDied","Data":"2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753"}
Jul 27 17:13:03.943516 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.943510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd" event={"ID":"dfc0ca6f-003d-4135-ac40-f195bd476e3b","Type":"ContainerDied","Data":"837b17491cac9f411c57d7aeccf4965eec9f0e54aa22326647f1b0b443665be2"}
Jul 27 17:13:03.951143 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.950995    2559 scope.go:117] "RemoveContainer" containerID="1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1"
Jul 27 17:13:03.951503 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:13:03.951482    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1\": container with ID starting with 1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1 not found: ID does not exist" containerID="1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1"
Jul 27 17:13:03.951609 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.951509    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1"} err="failed to get container status \"1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1\": rpc error: code = NotFound desc = could not find container \"1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1\": container with ID starting with 1be3bb0dd669cfc6eeb7a3af57c27df3ee4ee520ee3959c05e7cf201b33af6c1 not found: ID does not exist"
Jul 27 17:13:03.951609 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.951527    2559 scope.go:117] "RemoveContainer" containerID="2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753"
Jul 27 17:13:03.959084 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.959069    2559 scope.go:117] "RemoveContainer" containerID="2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753"
Jul 27 17:13:03.959355 ip-10-0-140-199 kubenswrapper[2559]: E0727 17:13:03.959325    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753\": container with ID starting with 2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753 not found: ID does not exist" containerID="2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753"
Jul 27 17:13:03.959440 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.959364    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753"} err="failed to get container status \"2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753\": rpc error: code = NotFound desc = could not find container \"2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753\": container with ID starting with 2fec42045aa6960d3c02d9954435ff5dcdf436dc3011a9d46f3adf7a9b723753 not found: ID does not exist"
Jul 27 17:13:03.966133 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.966110    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"]
Jul 27 17:13:03.970823 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.970803    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d4529a-6bbcb4c854-jvh7c"]
Jul 27 17:13:03.986788 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.986763    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"]
Jul 27 17:13:03.992057 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:03.992035    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d4529a-76fcf66c7c-zrkdd"]
Jul 27 17:13:04.736472 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:04.736432    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" path="/var/lib/kubelet/pods/4cb8cb50-a076-45fe-8436-e0c6a7ec05c5/volumes"
Jul 27 17:13:04.736973 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:04.736950    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc0ca6f-003d-4135-ac40-f195bd476e3b" path="/var/lib/kubelet/pods/dfc0ca6f-003d-4135-ac40-f195bd476e3b/volumes"
Jul 27 17:13:07.412410 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:07.412372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:07.830869 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:07.830782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:13.851512 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:13.851476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:13.920366 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:13.920331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:14.109057 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:14.108971    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:14.362693 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:14.362606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:20.503611 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:20.503581    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-556f76c959-zgl78_e3e69829-dc4a-46b8-9a5c-35cc02541ab9/manager/0.log"
Jul 27 17:13:20.615329 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:20.615274    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-g7k8c_13114416-d6a5-484c-99b2-674b29cf37ea/server/0.log"
Jul 27 17:13:20.737686 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:20.737653    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zjvv4_46619dce-d9fc-47b2-b8b7-d52ae6875b81/manager/0.log"
Jul 27 17:13:21.121042 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:21.121014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-rdlh6_e0f648ea-5889-4b82-96da-5d897d07f9b2/copy-manifests/0.log"
Jul 27 17:13:21.139686 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:21.139656    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-rdlh6_e0f648ea-5889-4b82-96da-5d897d07f9b2/manager/0.log"
Jul 27 17:13:21.601322 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:21.601290    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752830-rhkb5_370f01cc-ac79-4033-ac93-bde8166b53dc/cleanup/0.log"
Jul 27 17:13:21.714148 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:21.714122    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752845-jpbwq_1cc5fbda-d461-4376-a9f3-df29107724b2/cleanup/0.log"
Jul 27 17:13:24.552834 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:24.552798    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-98x86_56f0e6ee-5cb5-41a4-b05a-31af48de697f/discovery/0.log"
Jul 27 17:13:25.139990 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:25.139951    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qxpr8_2e603be2-8915-481b-9db0-4bf942836c39/payload-processing/0.log"
Jul 27 17:13:25.716051 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:25.716023    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nv7n2_4f52ebfd-4a7b-49b2-b3f1-0160e46f04d7/main/0.log"
Jul 27 17:13:25.986950 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:25.986870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f42vl4_df344889-1bb9-4d2c-8699-c8580ef4ef25/main/0.log"
Jul 27 17:13:31.778941 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:31.778901    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:31.880299 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:31.880251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log" limit=1024
Jul 27 17:13:34.974394 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:34.974307    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bbqvv_3357411f-3fd8-4a76-9d8d-e5af12e69251/global-pull-secret-syncer/0.log"
Jul 27 17:13:35.109061 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:35.109026    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k4mgt_1b16964c-e80c-4655-b921-22e9d67dc359/konnectivity-agent/0.log"
Jul 27 17:13:35.331673 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:35.331571    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-199.ec2.internal_83df169d31731b8be500e8b8ca40ce24/haproxy/0.log"
Jul 27 17:13:43.880404 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:43.880367    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s55g5_3c19f12f-07c3-470e-bb1c-cdf124c2cdf2/node-exporter/0.log"
Jul 27 17:13:43.916475 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:43.916437    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s55g5_3c19f12f-07c3-470e-bb1c-cdf124c2cdf2/kube-rbac-proxy/0.log"
Jul 27 17:13:43.946118 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:43.946089    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s55g5_3c19f12f-07c3-470e-bb1c-cdf124c2cdf2/init-textfile/0.log"
Jul 27 17:13:45.268691 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.268653    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"]
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.268972    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.268987    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.268998    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314e88b6-bd2d-4c6f-a880-019a6426709a" containerName="payload-pre-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269006    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="314e88b6-bd2d-4c6f-a880-019a6426709a" containerName="payload-pre-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269019    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ec2be3-167e-42e2-b6ca-011ebb2269fb" containerName="maas-api"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269026    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ec2be3-167e-42e2-b6ca-011ebb2269fb" containerName="maas-api"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269039    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc0ca6f-003d-4135-ac40-f195bd476e3b" containerName="maas-api"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269046    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc0ca6f-003d-4135-ac40-f195bd476e3b" containerName="maas-api"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269057    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8efe61-758f-430d-b383-e247eeb5c9b4" containerName="payload-pre-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269065    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8efe61-758f-430d-b383-e247eeb5c9b4" containerName="payload-pre-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269085    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269093    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269101    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269108    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269116    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" containerName="payload-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269125    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" containerName="payload-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269133    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb97743-545f-4162-a6a7-a7475a8f787f" containerName="payload-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269141    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb97743-545f-4162-a6a7-a7475a8f787f" containerName="payload-processing"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269152    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8006c3-9c72-4fa6-9b56-fa79358a37ae" containerName="maas-api"
Jul 27 17:13:45.269153 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269162    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8006c3-9c72-4fa6-9b56-fa79358a37ae" containerName="maas-api"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201bc181-2222-4e7f-a313-23ed9311a795" containerName="payload-pre-processing"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269180    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="201bc181-2222-4e7f-a313-23ed9311a795" containerName="payload-pre-processing"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269253    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="314e88b6-bd2d-4c6f-a880-019a6426709a" containerName="payload-pre-processing"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269263    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc0ca6f-003d-4135-ac40-f195bd476e3b" containerName="maas-api"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269298    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269305    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ec2be3-167e-42e2-b6ca-011ebb2269fb" containerName="maas-api"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269311    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269317    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="201bc181-2222-4e7f-a313-23ed9311a795" containerName="payload-pre-processing"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269323    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb97743-545f-4162-a6a7-a7475a8f787f" containerName="payload-processing"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269328    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8efe61-758f-430d-b383-e247eeb5c9b4" containerName="payload-pre-processing"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269333    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d769569c-c883-4423-9901-9791c69d55bf" containerName="istio-proxy"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269339    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8006c3-9c72-4fa6-9b56-fa79358a37ae" containerName="maas-api"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269345    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff07a830-9374-403f-909d-ce8e79eaee9e" containerName="istio-proxy"
Jul 27 17:13:45.269777 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.269350    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb8cb50-a076-45fe-8436-e0c6a7ec05c5" containerName="payload-processing"
Jul 27 17:13:45.273774 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.273758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.277305 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.277270    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddw25\"/\"kube-root-ca.crt\""
Jul 27 17:13:45.278484 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.278464    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ddw25\"/\"default-dockercfg-nqbtv\""
Jul 27 17:13:45.278589 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.278490    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddw25\"/\"openshift-service-ca.crt\""
Jul 27 17:13:45.285633 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.285612    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"]
Jul 27 17:13:45.368091 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.368055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-podres\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.368269 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.368109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmr4\" (UniqueName: \"kubernetes.io/projected/18a69dc7-fc71-4328-bdf7-b939ef4cb092-kube-api-access-6hmr4\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.368269 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.368192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-sys\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.368269 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.368215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-proc\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.368269 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.368233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-lib-modules\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469078 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-podres\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmr4\" (UniqueName: \"kubernetes.io/projected/18a69dc7-fc71-4328-bdf7-b939ef4cb092-kube-api-access-6hmr4\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-sys\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-proc\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-lib-modules\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-podres\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-sys\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469268 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-proc\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.469611 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.469320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18a69dc7-fc71-4328-bdf7-b939ef4cb092-lib-modules\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.481379 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.481351    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmr4\" (UniqueName: \"kubernetes.io/projected/18a69dc7-fc71-4328-bdf7-b939ef4cb092-kube-api-access-6hmr4\") pod \"perf-node-gather-daemonset-spn8p\" (UID: \"18a69dc7-fc71-4328-bdf7-b939ef4cb092\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.583673 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.583580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:45.715011 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:45.714985    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"]
Jul 27 17:13:45.717463 ip-10-0-140-199 kubenswrapper[2559]: W0727 17:13:45.717433    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod18a69dc7_fc71_4328_bdf7_b939ef4cb092.slice/crio-f3119ebdbd76a7e76412e77b0a470065097efc0a5a28892a1adf355f290729b1 WatchSource:0}: Error finding container f3119ebdbd76a7e76412e77b0a470065097efc0a5a28892a1adf355f290729b1: Status 404 returned error can't find the container with id f3119ebdbd76a7e76412e77b0a470065097efc0a5a28892a1adf355f290729b1
Jul 27 17:13:46.125403 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:46.125366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p" event={"ID":"18a69dc7-fc71-4328-bdf7-b939ef4cb092","Type":"ContainerStarted","Data":"fec9a507faddf45b8a0d91f48d69456e42e26f8fafa682fa21648019a7b8a1bf"}
Jul 27 17:13:46.125593 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:46.125420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p" event={"ID":"18a69dc7-fc71-4328-bdf7-b939ef4cb092","Type":"ContainerStarted","Data":"f3119ebdbd76a7e76412e77b0a470065097efc0a5a28892a1adf355f290729b1"}
Jul 27 17:13:46.125593 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:46.125481    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:46.147104 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:46.147054    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p" podStartSLOduration=1.147041045 podStartE2EDuration="1.147041045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:13:46.144221804 +0000 UTC m=+3426.045362262" watchObservedRunningTime="2026-07-27 17:13:46.147041045 +0000 UTC m=+3426.048181493"
Jul 27 17:13:49.221506 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:49.221479    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcfdm_419fc3b7-5ea7-46c6-8b4e-cc2a82958adc/dns/0.log"
Jul 27 17:13:49.266883 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:49.266847    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcfdm_419fc3b7-5ea7-46c6-8b4e-cc2a82958adc/kube-rbac-proxy/0.log"
Jul 27 17:13:49.458955 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:49.458923    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2k8f_bf283595-3a83-4674-af35-b7861330a709/dns-node-resolver/0.log"
Jul 27 17:13:50.201475 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:50.201448    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qmfdr_f795eac2-9fea-403e-ba0e-54c90483cc72/node-ca/0.log"
Jul 27 17:13:51.638350 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:51.638309    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-98x86_56f0e6ee-5cb5-41a4-b05a-31af48de697f/discovery/0.log"
Jul 27 17:13:51.963396 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:51.963365    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qxpr8_2e603be2-8915-481b-9db0-4bf942836c39/payload-processing/0.log"
Jul 27 17:13:52.138973 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:52.138944    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-spn8p"
Jul 27 17:13:52.914068 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:52.914015    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmzm2_cd1af325-26fe-4a63-8de9-f7ebe2f189a9/serve-healthcheck-canary/0.log"
Jul 27 17:13:53.642990 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:53.642955    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2m9g_a0eb2388-6a44-436c-ae6c-bf13dce0856e/kube-rbac-proxy/0.log"
Jul 27 17:13:53.666123 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:53.666099    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2m9g_a0eb2388-6a44-436c-ae6c-bf13dce0856e/exporter/0.log"
Jul 27 17:13:53.688788 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:53.688756    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z2m9g_a0eb2388-6a44-436c-ae6c-bf13dce0856e/extractor/0.log"
Jul 27 17:13:56.799840 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:56.799706    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-98x86_56f0e6ee-5cb5-41a4-b05a-31af48de697f/discovery/0.log"
Jul 27 17:13:56.961703 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:56.961666    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-556f76c959-zgl78_e3e69829-dc4a-46b8-9a5c-35cc02541ab9/manager/0.log"
Jul 27 17:13:56.978675 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:56.978645    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qxpr8_2e603be2-8915-481b-9db0-4bf942836c39/payload-processing/0.log"
Jul 27 17:13:57.003110 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:57.003084    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-g7k8c_13114416-d6a5-484c-99b2-674b29cf37ea/server/0.log"
Jul 27 17:13:57.136382 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:57.136298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zjvv4_46619dce-d9fc-47b2-b8b7-d52ae6875b81/manager/0.log"
Jul 27 17:13:57.381014 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:57.380980    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-rdlh6_e0f648ea-5889-4b82-96da-5d897d07f9b2/manager/0.log"
Jul 27 17:13:57.402544 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:57.402516    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-rdlh6_e0f648ea-5889-4b82-96da-5d897d07f9b2/copy-manifests/0.log"
Jul 27 17:13:58.659563 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:58.659526    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-98x86_56f0e6ee-5cb5-41a4-b05a-31af48de697f/discovery/0.log"
Jul 27 17:13:58.839338 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:58.839306    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cd69fdcb8-hk48f_bc0559f6-dc1d-48a7-a18a-2c8613af601d/manager/0.log"
Jul 27 17:13:58.840533 ip-10-0-140-199 kubenswrapper[2559]: I0727 17:13:58.840507    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-qxpr8_2e603be2-8915-481b-9db0-4bf942836c39/payload-processing/0.log"