Jul 29 19:27:42.847288 ip-10-0-138-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 19:27:42.847299 ip-10-0-138-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 19:27:42.847307 ip-10-0-138-139 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 19:27:42.847634 ip-10-0-138-139 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 19:27:53.065426 ip-10-0-138-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 19:27:53.065445 ip-10-0-138-139 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dbd3879b16844dc6991598f8a19ebbba --
Jul 29 19:29:52.408829 ip-10-0-138-139 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 19:29:52.863451 ip-10-0-138-139 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:29:52.863451 ip-10-0-138-139 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 19:29:52.863451 ip-10-0-138-139 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:29:52.863451 ip-10-0-138-139 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 19:29:52.863451 ip-10-0-138-139 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 19:29:52.864464 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.864384    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 19:29:52.868670 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868654    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:29:52.868670 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868670    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868674    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868677    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868680    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868682    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868685    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868687    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868691    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868694    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868697    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868699    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868702    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868705    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868708    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868710    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868713    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868716    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868718    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868721    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868729    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:29:52.868730 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868732    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868735    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868738    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868741    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868743    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868746    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868748    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868751    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868753    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868756    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868758    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868761    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868763    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868766    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868769    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868771    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868774    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868776    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868778    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868781    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:29:52.869219 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868783    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868785    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868788    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868790    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868794    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868798    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868800    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868804    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868806    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868809    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868812    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868814    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868817    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868819    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868822    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868825    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868828    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868830    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868833    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:29:52.869930 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868836    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868840    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868842    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868845    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868847    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868850    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868852    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868854    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868858    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868860    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868863    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868865    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868868    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868870    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868873    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868875    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868877    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868880    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868882    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868884    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:29:52.870498 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868887    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868889    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868891    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868894    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868898    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868901    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868904    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868908    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868910    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868913    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868915    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868918    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868920    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868922    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868925    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868927    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868930    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868932    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:29:52.871011 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.868935    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:29:52.871832 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871817    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:29:52.871832 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871832    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871835    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871838    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871841    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871844    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871849    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871853    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871856    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871861    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871864    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871867    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871871    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871875    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871878    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871880    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871883    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871886    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871888    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871891    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:29:52.871890 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871894    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871896    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871900    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871903    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871906    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871908    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871911    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871914    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871916    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871919    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871922    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871924    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871927    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871929    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871932    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871934    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871937    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871939    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871942    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:29:52.872389 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871944    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871947    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871949    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871951    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871954    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871956    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871959    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871961    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871963    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871966    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871968    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871970    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871973    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871976    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871978    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871981    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871984    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871986    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871989    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871992    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:29:52.872945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871994    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871997    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.871999    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872001    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872004    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872006    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872009    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872011    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872014    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872016    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872018    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872021    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872025    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872027    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872029    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872032    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872034    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872037    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872040    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872042    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:29:52.873428 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872045    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872047    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872049    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872052    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872055    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872059    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872062    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872066    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872069    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872072    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872074    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872076    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872079    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872081    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872084    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872086    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872089    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872091    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872094    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:29:52.873924 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872096    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872463    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872467    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872470    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872473    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872475    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872478    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872480    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872483    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872485    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872488    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872490    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872493    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872495    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872498    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872501    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872503    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872506    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872509    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872511    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:29:52.874484 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872516    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872519    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872521    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872524    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872526    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872529    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872531    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872533    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872536    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872538    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872558    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872562    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872566    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872569    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872572    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872574    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872577    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872579    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872582    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872585    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:29:52.874991 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872587    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872590    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872592    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872595    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872597    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872600    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872603    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872605    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872608    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872610    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872615    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872617    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872621    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872624    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872628    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872632    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872635    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872637    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872640    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872642    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:29:52.875472 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872645    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872647    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872649    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872652    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872654    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872656    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872659    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872661    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872664    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872666    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872669    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872672    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872674    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872676    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872679    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872681    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872684    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872686    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872690    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:29:52.875981 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872693    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872695    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872699    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872702    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872705    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872708    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872711    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872714    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872716    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872719    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872721    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872723    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872726    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872729    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872731    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872734    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872736    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872739    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872741    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872744    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:29:52.876438 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.872746    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872820    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872827    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872833    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872838    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872841    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872845    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872848    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872852    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872855    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872858    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872862    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872865    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872868    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872871    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872874    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872876    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872880    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872883    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872887    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872890    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872892    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872895    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872899    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 19:29:52.876941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872902    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872906    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872909    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872912    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872915    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872918    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872920    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872923    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872926    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872930    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872933    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872936    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872938    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872941    2555 flags.go:64] FLAG: --enable-server="true"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872944    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872950    2555 flags.go:64] FLAG: --event-burst="100"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872953    2555 flags.go:64] FLAG: --event-qps="50"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872956    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872959    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872962    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872965    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872969    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872972    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872975    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872978    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 19:29:52.877515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872982    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872985    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872988    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872991    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872994    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.872996    2555 flags.go:64] FLAG: --feature-gates=""
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873000    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873003    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873006    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873009    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873012    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873014    2555 flags.go:64] FLAG: --help="false"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873017    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-138-139.ec2.internal"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873020    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873023    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873026    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873029    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873032    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873035    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873038    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873041    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873044    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873047    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873050    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 19:29:52.878207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873053    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873055    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873058    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873061    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873064    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873067    2555 flags.go:64] FLAG: --lock-file=""
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873069    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873072    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873074    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873080    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873083    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873086    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873089    2555 flags.go:64] FLAG: --logging-format="text"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873091    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873094    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873097    2555 flags.go:64] FLAG: --manifest-url=""
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873100    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873104    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873106    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873110    2555 flags.go:64] FLAG: --max-pods="110"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873113    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873116    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873118    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873121    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873124    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 19:29:52.878813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873127    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873130    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873137    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873140    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873146    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873149    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873151    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873156    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873159    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873162    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873165    2555 flags.go:64] FLAG: --port="10250"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873167    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873170    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c84444322f97bdd6"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873174    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873176    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873179    2555 flags.go:64] FLAG: --register-node="true"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873182    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873186    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873189    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873192    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873195    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873198    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873201    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873204    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873207    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 19:29:52.879419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873209    2555 flags.go:64] FLAG: --runonce="false"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873212    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873215    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873218    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873221    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873224    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873227    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873230    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873233    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873235    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873238    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873241    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873245    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873248    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873251    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873253    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873259    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873261    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873264    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873268    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873270    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873273    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873276    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873278    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873281    2555 flags.go:64] FLAG: --v="2"
Jul 29 19:29:52.880075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873285    2555 flags.go:64] FLAG: --version="false"
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873289    2555 flags.go:64] FLAG: --vmodule=""
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873293    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.873296    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873405    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873409    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873412    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873414    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873417    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873420    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873423    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873426    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873428    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873432    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873435    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873438    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873441    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873443    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873446    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873450    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:29:52.880725 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873453    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873457    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873460    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873463    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873466    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873469    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873472    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873474    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873477    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873479    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873482    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873484    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873487    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873489    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873492    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873495    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873497    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873500    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873502    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:29:52.881206 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873506    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873509    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873512    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873516    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873520    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873523    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873526    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873528    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873530    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873533    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873536    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873538    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873562    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873567    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873570    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873573    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873576    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873578    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873581    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873583    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:29:52.881691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873586    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873589    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873591    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873593    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873596    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873598    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873601    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873603    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873607    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873609    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873611    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873614    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873616    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873619    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873621    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873624    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873626    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873629    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873631    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873634    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:29:52.882213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873636    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873639    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873641    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873643    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873649    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873652    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873654    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873657    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873659    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873662    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873664    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873667    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873669    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873672    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873674    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873677    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873679    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873682    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873684    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873687    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:29:52.882722 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873689    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873692    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873695    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.873697    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.874427    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.881653    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.881669    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.881688    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.881694    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881755    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881761    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881765    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881768    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881771    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:29:52.883213 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881774    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881777    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881780    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881783    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881786    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881789    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881792    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881794    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881797    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881800    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881803    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881806    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881808    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881811    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881814    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881816    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881819    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881822    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881824    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881828    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:29:52.883577 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881830    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881833    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881836    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881838    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881842    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881845    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881848    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881851    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881853    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881856    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881858    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881860    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881863    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881865    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881868    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881870    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881873    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881875    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881878    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881880    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:29:52.884055 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881883    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881885    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881887    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881890    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881892    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881895    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881897    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881899    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881903    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881908    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881912    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881915    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881918    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881921    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881923    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881926    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881929    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881932    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881934    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:29:52.884537 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881937    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881940    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881942    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881945    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881947    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881949    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881952    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881954    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881957    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881959    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881961    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881964    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881966    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881969    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881971    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881974    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881976    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881979    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881983    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881986    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:29:52.885124 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881989    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881992    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881994    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881997    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.881999    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882002    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882005    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882008    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882010    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882014    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882017    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882019    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882022    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882024    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882027    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:29:52.885620 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.882032    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882145    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882149    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882152    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882155    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882157    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882160    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882162    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882165    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882167    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882169    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882172    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882174    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882177    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882179    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882181    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882184    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882186    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882188    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882191    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882193    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 19:29:52.885996 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882196    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882198    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882201    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882203    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882206    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882209    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882212    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882215    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882217    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882220    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882222    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882225    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882227    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882230    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882232    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882235    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882237    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882239    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882241    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 19:29:52.886489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882244    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882247    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882251    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882253    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882256    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882258    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882261    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882263    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882266    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882268    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882271    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882273    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882277    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882280    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882283    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882286    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882288    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882291    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882294    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 19:29:52.886978 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882297    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882300    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882302    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882304    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882307    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882309    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882311    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882314    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882316    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882319    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882321    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882323    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882326    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882328    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882331    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882333    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882336    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882339    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882343    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 19:29:52.887436 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882345    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882348    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882350    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882353    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882355    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882358    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882360    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882362    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882365    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882367    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882370    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882372    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882375    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882378    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882380    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882383    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882385    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882387    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882390    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882392    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 19:29:52.887906 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882394    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:52.882397    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.882401    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.882498    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.884806    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.886424    2555 server.go:1019] "Starting client certificate rotation"
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.886528    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:29:52.888382 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.886581    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 19:29:52.890768 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.890754    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:29:52.891862 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.891848    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 19:29:52.901486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.901470    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 29 19:29:52.904305 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.904290    2555 log.go:25] "Validated CRI v1 image API"
Jul 29 19:29:52.906326 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.906308    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 19:29:52.911694 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.911674    2555 fs.go:135] Filesystem UUIDs: map[07502ffe-0f8a-478b-be8e-4107003d40e5:/dev/nvme0n1p3 4013b718-f2e1-4fc5-b163-f17616ea3e43:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 29 19:29:52.911755 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.911693    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 19:29:52.917898 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.917794    2555 manager.go:216] Machine: {Timestamp:2026-07-29 19:29:52.916484773 +0000 UTC m=+0.377011978 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101573 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25c2fd43237ee2a69be5f5f9be6a46 SystemUUID:ec25c2fd-4323-7ee2-a69b-e5f5f9be6a46 BootID:dbd3879b-1684-4dc6-9915-98f8a19ebbba Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:33:73:5f:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:33:73:5f:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:06:aa:51:6c:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 19:29:52.917898 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.917893    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 19:29:52.918008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.917993    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 19:29:52.921366 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.921343    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 19:29:52.921500 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.921368    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 19:29:52.922068 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.922056    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 19:29:52.922113 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.922071    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 19:29:52.922113 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.922078    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:29:52.922722 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.922712    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 19:29:52.923755 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.923745    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 19:29:52.925497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.925485    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:29:52.925635 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.925625    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 19:29:52.926417 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.926406    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 19:29:52.926455 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.926424    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 19:29:52.926455 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.926449    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 19:29:52.926509 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.926462    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 29 19:29:52.926509 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.926475    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 19:29:52.927204 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.927190    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:29:52.927267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.927212    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 19:29:52.929340 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.929324    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 19:29:52.931159 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.931138    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 19:29:52.931232 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.931183    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 19:29:52.931468 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.931448    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:29:52.933418 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933404    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933428    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933439    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933447    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933460    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933468    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933480    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933488    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 19:29:52.933497 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933499    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 19:29:52.933747 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933509    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 19:29:52.933747 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933529    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 19:29:52.933747 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.933538    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 19:29:52.936079 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.936067    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 19:29:52.936141 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.936082    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 19:29:52.937559 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.937456    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 19:29:52.937559 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.937473    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 19:29:52.940664 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.940645    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 19:29:52.941098 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.941085    2555 server.go:1268] "Started kubelet"
Jul 29 19:29:52.941208 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.941162    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 19:29:52.941422 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.941169    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 19:29:52.941503 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.941427    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 19:29:52.941901 ip-10-0-138-139 systemd[1]: Started Kubernetes Kubelet.
Jul 29 19:29:52.943803 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.943774    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 19:29:52.946675 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.946659    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 19:29:52.948320 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.948301    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 19:29:52.948506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.948491    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfb75"
Jul 29 19:29:52.948743 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.947735    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-139.ec2.internal.18c6d973a0ca37ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-139.ec2.internal,UID:ip-10-0-138-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-139.ec2.internal,},FirstTimestamp:2026-07-29 19:29:52.941053882 +0000 UTC m=+0.401581067,LastTimestamp:2026-07-29 19:29:52.941053882 +0000 UTC m=+0.401581067,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-139.ec2.internal,}"
Jul 29 19:29:52.953216 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.953189    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 19:29:52.953216 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.953214    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 19:29:52.953395 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.953320    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:52.953676 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.953655    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jfb75"
Jul 29 19:29:52.953867 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.953843    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 19:29:52.953867 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.953851    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 19:29:52.953867 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.953864    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 19:29:52.954073 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954056    2555 factory.go:55] Registering systemd factory
Jul 29 19:29:52.954073 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954074    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 29 19:29:52.954190 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954087    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 19:29:52.954190 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954098    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 19:29:52.954300 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954285    2555 factory.go:153] Registering CRI-O factory
Jul 29 19:29:52.954342 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954311    2555 factory.go:223] Registration of the crio container factory successfully
Jul 29 19:29:52.954372 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954361    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 19:29:52.954401 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954382    2555 factory.go:103] Registering Raw factory
Jul 29 19:29:52.954401 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.954393    2555 manager.go:1196] Started watching for new ooms in manager
Jul 29 19:29:52.954933 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.954904    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 19:29:52.955049 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:52.955027    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 19:29:52.955180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.955168    2555 manager.go:349] Starting recovery of all containers
Jul 29 19:29:52.966803 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.966779    2555 manager.go:354] Recovery completed
Jul 29 19:29:52.970644 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.970632    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:52.972345 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972328    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:52.972414 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972360    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:52.972414 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972373    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:52.972895 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972881    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 19:29:52.972944 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972896    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 19:29:52.972944 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972912    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:52.972944 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.972914    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 19:29:52.973567 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.973536    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:52.973604 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.973583    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:52.973604 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.973596    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:52.974639 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.974627    2555 policy_none.go:49] "None policy: Start"
Jul 29 19:29:52.974685 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.974644    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 19:29:52.974685 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.974655    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 19:29:52.975867 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:52.975857    2555 policy_none.go:47] "Start"
Jul 29 19:29:53.017089 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017068    2555 manager.go:337] "Starting Device Plugin manager"
Jul 29 19:29:53.017175 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.017123    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 19:29:53.017175 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017137    2555 server.go:87] "Starting device plugin registration server"
Jul 29 19:29:53.017405 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017393    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 19:29:53.017439 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017408    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 19:29:53.017522 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017508    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 19:29:53.017644 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017603    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 19:29:53.017644 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.017612    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 19:29:53.018187 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.018165    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 19:29:53.018277 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.018222    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.117828 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.117752    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:53.118799 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.118716    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:53.118799 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.118744    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:53.118799 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.118755    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:53.118799 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.118786    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.128512 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.128494    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.128579 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.128515    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-139.ec2.internal\": node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.144840 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.144825    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.148704 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.148667    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 19:29:53.150071 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.150055    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 19:29:53.150123 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.150092    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 19:29:53.150123 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.150115    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 19:29:53.150193 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.150158    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 19:29:53.154137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.154123    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:29:53.245444 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.245416    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.250603 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.250583    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"]
Jul 29 19:29:53.250667 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.250657    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:53.252379 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.252361    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:53.252456 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.252393    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:53.252456 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.252403    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:53.253826 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.253813    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:53.253938 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.253922    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.253992 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.253962    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:53.255094 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48b623d21ae99c3ed73e2aaafa3d4404-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal\" (UID: \"48b623d21ae99c3ed73e2aaafa3d4404\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.255136 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca69d5b906524dbafb4263c4750a6c45-config\") pod \"kube-apiserver-proxy-ip-10-0-138-139.ec2.internal\" (UID: \"ca69d5b906524dbafb4263c4750a6c45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.255136 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48b623d21ae99c3ed73e2aaafa3d4404-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal\" (UID: \"48b623d21ae99c3ed73e2aaafa3d4404\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.255666 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255649    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:53.255721 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255677    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:53.255721 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255691    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:53.255721 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255695    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:53.255721 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255712    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:53.255848 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.255724    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:53.256877 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.256859    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.256972 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.256885    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 19:29:53.257654 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.257635    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 19:29:53.257756 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.257666    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 19:29:53.257756 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.257676    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeHasSufficientPID"
Jul 29 19:29:53.286754 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.286732    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-139.ec2.internal\" not found" node="ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.290663 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.290648    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-139.ec2.internal\" not found" node="ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.346160 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.346143    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.355418 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.355399    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48b623d21ae99c3ed73e2aaafa3d4404-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal\" (UID: \"48b623d21ae99c3ed73e2aaafa3d4404\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.355535 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.355425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48b623d21ae99c3ed73e2aaafa3d4404-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal\" (UID: \"48b623d21ae99c3ed73e2aaafa3d4404\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.355535 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.355452    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca69d5b906524dbafb4263c4750a6c45-config\") pod \"kube-apiserver-proxy-ip-10-0-138-139.ec2.internal\" (UID: \"ca69d5b906524dbafb4263c4750a6c45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.355535 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.355490    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48b623d21ae99c3ed73e2aaafa3d4404-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal\" (UID: \"48b623d21ae99c3ed73e2aaafa3d4404\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.355535 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.355493    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca69d5b906524dbafb4263c4750a6c45-config\") pod \"kube-apiserver-proxy-ip-10-0-138-139.ec2.internal\" (UID: \"ca69d5b906524dbafb4263c4750a6c45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.355535 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.355495    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/48b623d21ae99c3ed73e2aaafa3d4404-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal\" (UID: \"48b623d21ae99c3ed73e2aaafa3d4404\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.446661 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.446567    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.547103 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.547065    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.589561 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.589516    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.594622 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.594606    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:53.647696 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.647667    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.748323 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.748233    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.848727 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.848697    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.885141 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.885118    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 19:29:53.885494 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.885251    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 19:29:53.949605 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:53.949579    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-139.ec2.internal\" not found"
Jul 29 19:29:53.958750 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.958636    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 19:24:54 +0000 UTC" deadline="2028-04-25 18:53:53.525888472 +0000 UTC"
Jul 29 19:29:53.958750 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.958691    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15263h23m59.567201637s"
Jul 29 19:29:53.971839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:53.971821    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:29:54.039327 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:54.039267    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:29:54.054006 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:54.053845    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:54.071576 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:54.071537    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:29:54.073409 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:54.073394    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal"
Jul 29 19:29:54.082315 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:54.082294    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 19:29:55.688374 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:55.688342    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b623d21ae99c3ed73e2aaafa3d4404.slice/crio-2345d3dbb0c53a1923b8f27b27dfe4ece70c3998900a2440a1e8a7fd7e2e4b9f WatchSource:0}: Error finding container 2345d3dbb0c53a1923b8f27b27dfe4ece70c3998900a2440a1e8a7fd7e2e4b9f: Status 404 returned error can't find the container with id 2345d3dbb0c53a1923b8f27b27dfe4ece70c3998900a2440a1e8a7fd7e2e4b9f
Jul 29 19:29:55.688655 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:55.688630    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca69d5b906524dbafb4263c4750a6c45.slice/crio-5a7804fa2939a5eb195c2e1f029e3397aa45fd4ebb602296fa2139d8c472e11b WatchSource:0}: Error finding container 5a7804fa2939a5eb195c2e1f029e3397aa45fd4ebb602296fa2139d8c472e11b: Status 404 returned error can't find the container with id 5a7804fa2939a5eb195c2e1f029e3397aa45fd4ebb602296fa2139d8c472e11b
Jul 29 19:29:55.694635 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:55.694619    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:29:56.029419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.029340    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:29:56.303026 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.302943    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 19:29:56.429857 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.429824    2555 apiserver.go:52] "Watching apiserver"
Jul 29 19:29:56.434256 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.434239    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 19:29:56.434640 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.434621    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-crpff","openshift-multus/network-metrics-daemon-g24sx","openshift-network-diagnostics/network-check-target-mck7p","openshift-ovn-kubernetes/ovnkube-node-rg625","kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj","openshift-cluster-node-tuning-operator/tuned-9jgs2","openshift-dns/node-resolver-6g6hg","openshift-image-registry/node-ca-hw7vt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal","openshift-network-operator/iptables-alerter-hmhxl","kube-system/global-pull-secret-syncer-gr9w2","kube-system/konnectivity-agent-h27gc","openshift-multus/multus-additional-cni-plugins-qp955"]
Jul 29 19:29:56.438466 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.438449    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.440221 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.440191    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.440351 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.440192    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.440438 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.440422    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8qhs8\""
Jul 29 19:29:56.440552 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.440536    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.442432 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.442409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.442557 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.442458    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-66bzf\""
Jul 29 19:29:56.442557 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.442480    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.442557 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.442458    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 19:29:56.443412 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.443398    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:29:56.443499 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.443453    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:29:56.447218 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.447202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.449090 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.449037    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 19:29:56.449230 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.449213    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 19:29:56.449230 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.449222    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vj82\""
Jul 29 19:29:56.450023 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.450002    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 19:29:56.450108 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.450026    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.450760 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.450742    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.450927 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.450906    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.451429    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.452633    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.453445    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.453830    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8r684\""
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.453907    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.453974    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 19:29:56.454134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.454001    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 19:29:56.454486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.454247    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.454486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.454413    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 19:29:56.454486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.454481    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.454661 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.454644    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tkshp\""
Jul 29 19:29:56.454963 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.454945    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.455202 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.455181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.456470 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.456453    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 19:29:56.457271 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.457252    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 19:29:56.457554 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.457536    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2fkcr\""
Jul 29 19:29:56.457648 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.457598    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.457702 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.457539    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.458327 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.458308    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.460288 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.460011    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 19:29:56.460288 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.460018    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 19:29:56.460288 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.460053    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7nhj\""
Jul 29 19:29:56.460505 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.460371    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:56.460505 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.460433    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:29:56.462637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.462618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.464383 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.464364    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.464745    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465024    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sr9g4\""
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-var-lib-kubelet\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465702    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-modprobe-d\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465726    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-systemd\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-slash\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465808    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c822540-948e-4f1e-8189-00f4c8bff2b8-host\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465861    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465908    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-node-log\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-env-overrides\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.465979    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-registration-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-tuned\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466031    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2612c95d-e872-4e72-8fb6-cd7714290445-host-slash\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5ls\" (UniqueName: \"kubernetes.io/projected/2612c95d-e872-4e72-8fb6-cd7714290445-kube-api-access-6h5ls\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.466069 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-run-netns\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-cni-bin\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466119    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-ovn\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466141    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-os-release\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysctl-conf\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-sys\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-systemd-units\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-systemd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466335    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-cni-netd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-kubelet\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466452    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-device-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466480    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-etc-selinux\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466505    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbk2w\" (UniqueName: \"kubernetes.io/projected/0c822540-948e-4f1e-8189-00f4c8bff2b8-kube-api-access-fbk2w\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.466942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2612c95d-e872-4e72-8fb6-cd7714290445-iptables-alerter-script\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c633756-10c1-43d4-9dc0-b722ffa6deb0-agent-certs\") pod \"konnectivity-agent-h27gc\" (UID: \"1c633756-10c1-43d4-9dc0-b722ffa6deb0\") " pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466616    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-lib-modules\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466641    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysconfig\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466670    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d113c2-5b6f-4091-bb1b-111b923d7dad-tmp\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466715    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczlc\" (UniqueName: \"kubernetes.io/projected/af9419a1-7ae3-4842-86ef-ee9498a42799-kube-api-access-vczlc\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466736    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5qmkt\""
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466947    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.466990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-etc-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc12525f-9d1b-4f9b-b066-4781885c641d-ovn-node-metrics-cert\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-sys-fs\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c822540-948e-4f1e-8189-00f4c8bff2b8-serviceca\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467132    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrl8\" (UniqueName: \"kubernetes.io/projected/bfb04e9c-60d5-47d2-af30-45bd0cf67021-kube-api-access-9lrl8\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6pp\" (UniqueName: \"kubernetes.io/projected/c2d113c2-5b6f-4091-bb1b-111b923d7dad-kube-api-access-sg6pp\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.467609 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-ovnkube-script-lib\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467259    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c633756-10c1-43d4-9dc0-b722ffa6deb0-konnectivity-ca\") pod \"konnectivity-agent-h27gc\" (UID: \"1c633756-10c1-43d4-9dc0-b722ffa6deb0\") " pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-host\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467313    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysctl-d\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467379    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467394    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-run\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-log-socket\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467466    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-ovnkube-config\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467488    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-system-cni-dir\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.467437    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-kubernetes\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467558    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-var-lib-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467599    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxmd\" (UniqueName: \"kubernetes.io/projected/fc12525f-9d1b-4f9b-b066-4781885c641d-kube-api-access-vxxmd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cnibin\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.468207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.467693    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-socket-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.469870 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.469853    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 19:29:56.491037 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.491016    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fmdh"
Jul 29 19:29:56.500682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.500657    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9fmdh"
Jul 29 19:29:56.557396 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.557333    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 19:29:56.568448 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568418    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c633756-10c1-43d4-9dc0-b722ffa6deb0-konnectivity-ca\") pod \"konnectivity-agent-h27gc\" (UID: \"1c633756-10c1-43d4-9dc0-b722ffa6deb0\") " pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.568568 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-cni-multus\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.568568 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-host\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568568 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysctl-d\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568568 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-run\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-log-socket\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568626    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-host\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-ovnkube-config\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568688    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-system-cni-dir\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568697    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-run\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-kubernetes\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-var-lib-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-kubernetes\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-system-cni-dir\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxmd\" (UniqueName: \"kubernetes.io/projected/fc12525f-9d1b-4f9b-b066-4781885c641d-kube-api-access-vxxmd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568828    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-var-lib-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568689    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-log-socket\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568700    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysctl-d\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568850    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cnibin\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.568902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-socket-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-cni-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568959    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-cni-bin\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cnibin\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.568982    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4adba-8f03-4c88-96d3-cfc9449182f7-tmp-dir\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-var-lib-kubelet\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569114    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-modprobe-d\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-socket-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569132    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c633756-10c1-43d4-9dc0-b722ffa6deb0-konnectivity-ca\") pod \"konnectivity-agent-h27gc\" (UID: \"1c633756-10c1-43d4-9dc0-b722ffa6deb0\") " pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569137    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-var-lib-kubelet\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569138    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-systemd\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569189    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-systemd\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569198    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-slash\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569207    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-modprobe-d\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c822540-948e-4f1e-8189-00f4c8bff2b8-host\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569258    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-slash\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.569564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-ovnkube-config\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569313    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fd2995c-1986-43db-b5ff-dd46e220a926-cni-binary-copy\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569378    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569398    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-hostroot\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569404    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569324    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c822540-948e-4f1e-8189-00f4c8bff2b8-host\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569466    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-node-log\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569489    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-env-overrides\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569515    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-registration-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-node-log\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-registration-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-system-cni-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569623    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-tuned\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.570473 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2612c95d-e872-4e72-8fb6-cd7714290445-host-slash\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569652    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5ls\" (UniqueName: \"kubernetes.io/projected/2612c95d-e872-4e72-8fb6-cd7714290445-kube-api-access-6h5ls\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-run-netns\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-cni-bin\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2612c95d-e872-4e72-8fb6-cd7714290445-host-slash\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-multus-certs\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-ovn\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-os-release\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-daemon-config\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0d368bf-30b8-4d18-a17f-306eee453ddd-kubelet-config\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569898    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0d368bf-30b8-4d18-a17f-306eee453ddd-dbus\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.569989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-env-overrides\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570024    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-ovn\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570026    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-run-netns\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-cni-bin\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.571263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bfb04e9c-60d5-47d2-af30-45bd0cf67021-os-release\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d4adba-8f03-4c88-96d3-cfc9449182f7-hosts-file\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysctl-conf\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-sys\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570451    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-sys\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570553    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysctl-conf\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-systemd-units\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-systemd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-cni-netd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570685    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-kubelet\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-run-systemd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-device-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570781    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-systemd-units\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570783    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.572160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-device-dir\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570812    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-kubelet\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-etc-selinux\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbk2w\" (UniqueName: \"kubernetes.io/projected/0c822540-948e-4f1e-8189-00f4c8bff2b8-kube-api-access-fbk2w\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570860    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-kubelet\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2612c95d-e872-4e72-8fb6-cd7714290445-iptables-alerter-script\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c633756-10c1-43d4-9dc0-b722ffa6deb0-agent-certs\") pod \"konnectivity-agent-h27gc\" (UID: \"1c633756-10c1-43d4-9dc0-b722ffa6deb0\") " pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhsx\" (UniqueName: \"kubernetes.io/projected/16d4adba-8f03-4c88-96d3-cfc9449182f7-kube-api-access-mjhsx\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.570982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-lib-modules\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysconfig\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d113c2-5b6f-4091-bb1b-111b923d7dad-tmp\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-cni-netd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-sysconfig\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-netns\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-etc-selinux\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.572945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571149    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-os-release\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczlc\" (UniqueName: \"kubernetes.io/projected/af9419a1-7ae3-4842-86ef-ee9498a42799-kube-api-access-vczlc\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571216    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2d113c2-5b6f-4091-bb1b-111b923d7dad-lib-modules\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571244    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-conf-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-etc-kubernetes\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-525m2\" (UniqueName: \"kubernetes.io/projected/5fd2995c-1986-43db-b5ff-dd46e220a926-kube-api-access-525m2\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-etc-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc12525f-9d1b-4f9b-b066-4781885c641d-ovn-node-metrics-cert\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571413    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-sys-fs\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c822540-948e-4f1e-8189-00f4c8bff2b8-serviceca\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-k8s-cni-cncf-io\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571480    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrl8\" (UniqueName: \"kubernetes.io/projected/bfb04e9c-60d5-47d2-af30-45bd0cf67021-kube-api-access-9lrl8\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-cnibin\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-socket-dir-parent\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.573748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571599    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xnt\" (UniqueName: \"kubernetes.io/projected/1fb6effe-4bf8-46eb-b57c-414367796ec5-kube-api-access-p6xnt\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:56.575123 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.575094    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-run-ovn-kubernetes\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.575262 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.575242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.575329 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.575281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af9419a1-7ae3-4842-86ef-ee9498a42799-sys-fs\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.575759 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.575735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c822540-948e-4f1e-8189-00f4c8bff2b8-serviceca\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.575851 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.575772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bfb04e9c-60d5-47d2-af30-45bd0cf67021-cni-binary-copy\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.575968 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.575920    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc12525f-9d1b-4f9b-b066-4781885c641d-etc-openvswitch\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.575968 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.571479    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2612c95d-e872-4e72-8fb6-cd7714290445-iptables-alerter-script\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.576087 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.576072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6pp\" (UniqueName: \"kubernetes.io/projected/c2d113c2-5b6f-4091-bb1b-111b923d7dad-kube-api-access-sg6pp\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.576148 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.576117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-ovnkube-script-lib\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.576667 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.576564    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c633756-10c1-43d4-9dc0-b722ffa6deb0-agent-certs\") pod \"konnectivity-agent-h27gc\" (UID: \"1c633756-10c1-43d4-9dc0-b722ffa6deb0\") " pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.576742 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.576695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc12525f-9d1b-4f9b-b066-4781885c641d-ovnkube-script-lib\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.579015 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.577628    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc12525f-9d1b-4f9b-b066-4781885c641d-ovn-node-metrics-cert\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.582436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.582414    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2d113c2-5b6f-4091-bb1b-111b923d7dad-etc-tuned\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.584177 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.584151    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxmd\" (UniqueName: \"kubernetes.io/projected/fc12525f-9d1b-4f9b-b066-4781885c641d-kube-api-access-vxxmd\") pod \"ovnkube-node-rg625\" (UID: \"fc12525f-9d1b-4f9b-b066-4781885c641d\") " pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.584409 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.584389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2d113c2-5b6f-4091-bb1b-111b923d7dad-tmp\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.585090 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.585015    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:29:56.585090 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.585042    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:29:56.585090 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.585056    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:29:56.585090 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.585117    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:29:57.085097443 +0000 UTC m=+3.043592319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:29:56.587681 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.587612    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6pp\" (UniqueName: \"kubernetes.io/projected/c2d113c2-5b6f-4091-bb1b-111b923d7dad-kube-api-access-sg6pp\") pod \"tuned-9jgs2\" (UID: \"c2d113c2-5b6f-4091-bb1b-111b923d7dad\") " pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.589332 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.588952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczlc\" (UniqueName: \"kubernetes.io/projected/af9419a1-7ae3-4842-86ef-ee9498a42799-kube-api-access-vczlc\") pod \"aws-ebs-csi-driver-node-dd9pj\" (UID: \"af9419a1-7ae3-4842-86ef-ee9498a42799\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.590054 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.589939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5ls\" (UniqueName: \"kubernetes.io/projected/2612c95d-e872-4e72-8fb6-cd7714290445-kube-api-access-6h5ls\") pod \"iptables-alerter-hmhxl\" (UID: \"2612c95d-e872-4e72-8fb6-cd7714290445\") " pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.592613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.590837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrl8\" (UniqueName: \"kubernetes.io/projected/bfb04e9c-60d5-47d2-af30-45bd0cf67021-kube-api-access-9lrl8\") pod \"multus-additional-cni-plugins-qp955\" (UID: \"bfb04e9c-60d5-47d2-af30-45bd0cf67021\") " pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.592613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.591232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbk2w\" (UniqueName: \"kubernetes.io/projected/0c822540-948e-4f1e-8189-00f4c8bff2b8-kube-api-access-fbk2w\") pod \"node-ca-hw7vt\" (UID: \"0c822540-948e-4f1e-8189-00f4c8bff2b8\") " pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.658900 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.658839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal" event={"ID":"ca69d5b906524dbafb4263c4750a6c45","Type":"ContainerStarted","Data":"5a7804fa2939a5eb195c2e1f029e3397aa45fd4ebb602296fa2139d8c472e11b"}
Jul 29 19:29:56.663323 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.663294    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal" event={"ID":"48b623d21ae99c3ed73e2aaafa3d4404","Type":"ContainerStarted","Data":"2345d3dbb0c53a1923b8f27b27dfe4ece70c3998900a2440a1e8a7fd7e2e4b9f"}
Jul 29 19:29:56.676953 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.676920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-k8s-cni-cncf-io\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677087 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.676963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-cnibin\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677087 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.676998    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-socket-dir-parent\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677087 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xnt\" (UniqueName: \"kubernetes.io/projected/1fb6effe-4bf8-46eb-b57c-414367796ec5-kube-api-access-p6xnt\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:56.677087 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677054    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-cni-multus\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-cni-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677114    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-cni-bin\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677138    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4adba-8f03-4c88-96d3-cfc9449182f7-tmp-dir\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.677248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fd2995c-1986-43db-b5ff-dd46e220a926-cni-binary-copy\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-hostroot\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677216    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-system-cni-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-multus-certs\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-daemon-config\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0d368bf-30b8-4d18-a17f-306eee453ddd-kubelet-config\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677353    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0d368bf-30b8-4d18-a17f-306eee453ddd-dbus\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d4adba-8f03-4c88-96d3-cfc9449182f7-hosts-file\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677429    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-kubelet\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.677805 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhsx\" (UniqueName: \"kubernetes.io/projected/16d4adba-8f03-4c88-96d3-cfc9449182f7-kube-api-access-mjhsx\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.677805 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-netns\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677805 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677560    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-os-release\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677805 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677604    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-conf-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677805 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-etc-kubernetes\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.677805 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-525m2\" (UniqueName: \"kubernetes.io/projected/5fd2995c-1986-43db-b5ff-dd46e220a926-kube-api-access-525m2\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678017 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.677957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-k8s-cni-cncf-io\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678017 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678005    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-cnibin\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678096 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-socket-dir-parent\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678228 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-cni-multus\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678329 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678305    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-cni-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678365    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-cni-bin\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678683    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-var-lib-kubelet\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.678800 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.678719    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:29:56.678863 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.678807    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:29:57.178767838 +0000 UTC m=+3.137262711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:29:56.678929 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.678896    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:29:56.678988 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:56.678971    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:29:57.178953058 +0000 UTC m=+3.137447949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:29:56.679049 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.678996    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-system-cni-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679049 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0d368bf-30b8-4d18-a17f-306eee453ddd-dbus\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.679049 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679041    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-multus-certs\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679200 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-os-release\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679200 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679154    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-host-run-netns\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679198    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16d4adba-8f03-4c88-96d3-cfc9449182f7-hosts-file\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.679292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679206    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fd2995c-1986-43db-b5ff-dd46e220a926-cni-binary-copy\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679245    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0d368bf-30b8-4d18-a17f-306eee453ddd-kubelet-config\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:56.679292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679266    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4adba-8f03-4c88-96d3-cfc9449182f7-tmp-dir\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.679292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-hostroot\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679513 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679293    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-conf-dir\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679513 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679304    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fd2995c-1986-43db-b5ff-dd46e220a926-etc-kubernetes\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.679615 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.679511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5fd2995c-1986-43db-b5ff-dd46e220a926-multus-daemon-config\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.689182 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.689156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-525m2\" (UniqueName: \"kubernetes.io/projected/5fd2995c-1986-43db-b5ff-dd46e220a926-kube-api-access-525m2\") pod \"multus-crpff\" (UID: \"5fd2995c-1986-43db-b5ff-dd46e220a926\") " pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.689717 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.689698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhsx\" (UniqueName: \"kubernetes.io/projected/16d4adba-8f03-4c88-96d3-cfc9449182f7-kube-api-access-mjhsx\") pod \"node-resolver-6g6hg\" (UID: \"16d4adba-8f03-4c88-96d3-cfc9449182f7\") " pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.691505 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.691477    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xnt\" (UniqueName: \"kubernetes.io/projected/1fb6effe-4bf8-46eb-b57c-414367796ec5-kube-api-access-p6xnt\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:56.749031 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.748998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9jgs2"
Jul 29 19:29:56.759434 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.759406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hmhxl"
Jul 29 19:29:56.769690 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.769667    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:29:56.773845 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.773819    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2612c95d_e872_4e72_8fb6_cd7714290445.slice/crio-c546dcd46d010d28fcf3e4452b83df2ea8318c19f30f5b29ade5417ebcf134da WatchSource:0}: Error finding container c546dcd46d010d28fcf3e4452b83df2ea8318c19f30f5b29ade5417ebcf134da: Status 404 returned error can't find the container with id c546dcd46d010d28fcf3e4452b83df2ea8318c19f30f5b29ade5417ebcf134da
Jul 29 19:29:56.773951 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.773916    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qp955"
Jul 29 19:29:56.780282 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.780261    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj"
Jul 29 19:29:56.786003 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.785965    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc12525f_9d1b_4f9b_b066_4781885c641d.slice/crio-373f8b4e3de7aa8db4fb97f12ca58056193136bc1c827966e9fd3f4254a74148 WatchSource:0}: Error finding container 373f8b4e3de7aa8db4fb97f12ca58056193136bc1c827966e9fd3f4254a74148: Status 404 returned error can't find the container with id 373f8b4e3de7aa8db4fb97f12ca58056193136bc1c827966e9fd3f4254a74148
Jul 29 19:29:56.788226 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.788197    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hw7vt"
Jul 29 19:29:56.794648 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.794626    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9419a1_7ae3_4842_86ef_ee9498a42799.slice/crio-c8c52364281752d0ecb6c3615218a192de1c52d8de5774ce66a7397e6fd13008 WatchSource:0}: Error finding container c8c52364281752d0ecb6c3615218a192de1c52d8de5774ce66a7397e6fd13008: Status 404 returned error can't find the container with id c8c52364281752d0ecb6c3615218a192de1c52d8de5774ce66a7397e6fd13008
Jul 29 19:29:56.795865 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.795849    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:29:56.798612 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.798568    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb04e9c_60d5_47d2_af30_45bd0cf67021.slice/crio-3dba1fa5a23c7abd05cfc9dc2456ac71b5b34edb4f8aa7c262f2ca571d730e59 WatchSource:0}: Error finding container 3dba1fa5a23c7abd05cfc9dc2456ac71b5b34edb4f8aa7c262f2ca571d730e59: Status 404 returned error can't find the container with id 3dba1fa5a23c7abd05cfc9dc2456ac71b5b34edb4f8aa7c262f2ca571d730e59
Jul 29 19:29:56.801935 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.801915    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6g6hg"
Jul 29 19:29:56.805272 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.805247    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c822540_948e_4f1e_8189_00f4c8bff2b8.slice/crio-5eb8d4816817fbaa1464b6d1f6269481cb226e018aa37c2576c1af8f41302663 WatchSource:0}: Error finding container 5eb8d4816817fbaa1464b6d1f6269481cb226e018aa37c2576c1af8f41302663: Status 404 returned error can't find the container with id 5eb8d4816817fbaa1464b6d1f6269481cb226e018aa37c2576c1af8f41302663
Jul 29 19:29:56.807915 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:56.807866    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-crpff"
Jul 29 19:29:56.817168 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.817142    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d4adba_8f03_4c88_96d3_cfc9449182f7.slice/crio-e196a8ef9b64b2dd4c98b5d085ad73674473c351ff88f8061c10099868bc42bf WatchSource:0}: Error finding container e196a8ef9b64b2dd4c98b5d085ad73674473c351ff88f8061c10099868bc42bf: Status 404 returned error can't find the container with id e196a8ef9b64b2dd4c98b5d085ad73674473c351ff88f8061c10099868bc42bf
Jul 29 19:29:56.822457 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:29:56.822424    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd2995c_1986_43db_b5ff_dd46e220a926.slice/crio-b9bc8a4ac53c8270e7d0c302fb6ba651e9c563219b8619ea44d797cc1858b56a WatchSource:0}: Error finding container b9bc8a4ac53c8270e7d0c302fb6ba651e9c563219b8619ea44d797cc1858b56a: Status 404 returned error can't find the container with id b9bc8a4ac53c8270e7d0c302fb6ba651e9c563219b8619ea44d797cc1858b56a
Jul 29 19:29:57.181843 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.181758    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:57.181843 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.181807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:29:57.181843 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.181830    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.181930    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.181935    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.181947    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.181966    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.181978    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.181994    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:29:58.18197623 +0000 UTC m=+4.140471105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.182012    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:29:58.18200247 +0000 UTC m=+4.140497355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:29:57.182096 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:57.182027    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:29:58.182019665 +0000 UTC m=+4.140514555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:29:57.503590 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.503535    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 19:24:56 +0000 UTC" deadline="2028-04-22 07:43:01.216111598 +0000 UTC"
Jul 29 19:29:57.503590 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.503573    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15180h13m3.712543047s"
Jul 29 19:29:57.666602 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.666462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-crpff" event={"ID":"5fd2995c-1986-43db-b5ff-dd46e220a926","Type":"ContainerStarted","Data":"b9bc8a4ac53c8270e7d0c302fb6ba651e9c563219b8619ea44d797cc1858b56a"}
Jul 29 19:29:57.668096 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.668027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h27gc" event={"ID":"1c633756-10c1-43d4-9dc0-b722ffa6deb0","Type":"ContainerStarted","Data":"46e36e1f2596d28b88ad310dcac34e7ea4ba89f67376e1010cc3f6ee48645d4d"}
Jul 29 19:29:57.669846 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.669773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj" event={"ID":"af9419a1-7ae3-4842-86ef-ee9498a42799","Type":"ContainerStarted","Data":"c8c52364281752d0ecb6c3615218a192de1c52d8de5774ce66a7397e6fd13008"}
Jul 29 19:29:57.671231 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.671170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerStarted","Data":"3dba1fa5a23c7abd05cfc9dc2456ac71b5b34edb4f8aa7c262f2ca571d730e59"}
Jul 29 19:29:57.673204 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.673126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6g6hg" event={"ID":"16d4adba-8f03-4c88-96d3-cfc9449182f7","Type":"ContainerStarted","Data":"e196a8ef9b64b2dd4c98b5d085ad73674473c351ff88f8061c10099868bc42bf"}
Jul 29 19:29:57.674533 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.674483    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hw7vt" event={"ID":"0c822540-948e-4f1e-8189-00f4c8bff2b8","Type":"ContainerStarted","Data":"5eb8d4816817fbaa1464b6d1f6269481cb226e018aa37c2576c1af8f41302663"}
Jul 29 19:29:57.675919 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.675838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"373f8b4e3de7aa8db4fb97f12ca58056193136bc1c827966e9fd3f4254a74148"}
Jul 29 19:29:57.677043 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.676978    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hmhxl" event={"ID":"2612c95d-e872-4e72-8fb6-cd7714290445","Type":"ContainerStarted","Data":"c546dcd46d010d28fcf3e4452b83df2ea8318c19f30f5b29ade5417ebcf134da"}
Jul 29 19:29:57.678353 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:57.678329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9jgs2" event={"ID":"c2d113c2-5b6f-4091-bb1b-111b923d7dad","Type":"ContainerStarted","Data":"78470f611eac9c0de1f967175f60e27cab0f046933f404727e334953c7a1a0df"}
Jul 29 19:29:58.193981 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.193943    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:58.194164 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.194013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:29:58.194164 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.194048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:58.194291 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.194192    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:29:58.194291 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.194256    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:30:00.194238373 +0000 UTC m=+6.152733247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:29:58.195603 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.194981    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:29:58.195603 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.195004    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:29:58.195603 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.195017    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:29:58.195603 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.195069    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:00.19505216 +0000 UTC m=+6.153547046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:29:58.195603 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.195132    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:29:58.195603 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.195166    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:00.195154178 +0000 UTC m=+6.153649064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:29:58.504215 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.504109    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 19:24:56 +0000 UTC" deadline="2028-03-21 09:39:02.163571792 +0000 UTC"
Jul 29 19:29:58.504215 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.504172    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14414h9m3.659403752s"
Jul 29 19:29:58.656479 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.655678    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:29:58.656479 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.655819    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:29:58.656479 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.656211    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:29:58.656479 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.656307    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:29:58.656479 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:29:58.656370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:29:58.656479 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:29:58.656435    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:00.211819 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:00.211786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:00.212284 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:00.211835    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:00.212284 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:00.211887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:00.212284 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212016    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:00.212284 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212072    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:04.212053927 +0000 UTC m=+10.170548802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:00.212506 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212477    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:30:00.212506 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212493    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:30:00.212506 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212506    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:00.212682 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212549    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:04.212534179 +0000 UTC m=+10.171029068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:00.212682 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212628    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:00.212682 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.212662    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:30:04.212651063 +0000 UTC m=+10.171145938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:00.657284 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:00.657207    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:00.657443 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.657347    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:00.657794 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:00.657772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:00.657917 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.657866    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:00.657917 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:00.657912    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:00.658013 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:00.657984    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:02.652945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:02.652861    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:02.652945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:02.652889    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:02.653388 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:02.652871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:02.653388 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:02.653007    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:02.653388 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:02.653117    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:02.653388 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:02.653199    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:04.248808 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:04.248759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:04.249276 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:04.248817    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:04.249276 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:04.248875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:04.249276 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249012    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:04.249276 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249077    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:12.249056888 +0000 UTC m=+18.207551762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:04.249519 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249485    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:30:04.249519 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249507    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:30:04.249519 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249520    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:04.249697 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249564    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:12.249549755 +0000 UTC m=+18.208044650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:04.249697 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249647    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:04.249697 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.249683    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:30:12.249671649 +0000 UTC m=+18.208166523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:04.654348 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:04.653623    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:04.654348 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.653761    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:04.654348 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:04.654140    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:04.654348 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.654222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:04.654348 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:04.654270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:04.654348 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:04.654318    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:06.654186 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:06.653693    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:06.654186 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:06.653693    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:06.654186 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:06.653827    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:06.654186 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:06.653846    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:06.654186 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:06.653936    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:06.654186 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:06.654040    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:08.652901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:08.652868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:08.652901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:08.652901    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:08.653439 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:08.652870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:08.653439 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:08.652998    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:08.653439 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:08.653075    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:08.653439 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:08.653142    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:10.655858 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:10.655822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:10.655858 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:10.655852    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:10.656414 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:10.655962    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:10.656414 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:10.655998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:10.656414 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:10.656129    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:10.656414 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:10.656224    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:12.309924 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:12.309885    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:12.309924 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:12.309931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:12.309976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310073    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310083    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310098    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310111    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310122    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310139    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:28.310122706 +0000 UTC m=+34.268617581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310160    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:28.310147533 +0000 UTC m=+34.268642422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:12.310489 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.310177    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:30:28.310168535 +0000 UTC m=+34.268663408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:12.653486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:12.653405    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:12.653486 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:12.653434    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:12.653813 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.653527    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:12.653813 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.653615    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:12.653813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:12.653676    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:12.653813 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:12.653762    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:14.654012 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:14.653977    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:14.654474 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:14.654090    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:14.654474 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:14.654089    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:14.654474 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:14.654219    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:14.654474 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:14.654256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:14.654474 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:14.654410    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:16.653333 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.653081    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:16.653836 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:16.653417    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:16.653836 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.653761    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:16.653961 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:16.653862    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:16.653961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.653916    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:16.654059 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:16.653979    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:16.724033 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.723188    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6g6hg" event={"ID":"16d4adba-8f03-4c88-96d3-cfc9449182f7","Type":"ContainerStarted","Data":"424adfe4bd31baad5a761fb0d6d851a186088d869f0529261328c12bdb3c557b"}
Jul 29 19:30:16.725295 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.725258    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hw7vt" event={"ID":"0c822540-948e-4f1e-8189-00f4c8bff2b8","Type":"ContainerStarted","Data":"fcb129e0ed58a2fc62b9f311e7ac045321a90e52dd8b3ab744d7561e8b626a65"}
Jul 29 19:30:16.729655 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.729633    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:30:16.730253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.729999    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc12525f-9d1b-4f9b-b066-4781885c641d" containerID="eaef15e0e6660ce8747884966a7eb0a57de7bfd0aafe17b0bacb89cac0757367" exitCode=1
Jul 29 19:30:16.730253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.730056    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"53ef461ca0668b822372200d4205fa0ed55c9449becec2f3f0b51b3db94c74ce"}
Jul 29 19:30:16.730253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.730109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"89795e604becfbb9f116d1e4dd7e1afdd4582a9df6b0833da59dc5ecae83cb56"}
Jul 29 19:30:16.730253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.730141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"86789f622bae93e434b7d64f1d581d69d74c0ea47a1f35892464ff464bc226d1"}
Jul 29 19:30:16.730253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.730173    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerDied","Data":"eaef15e0e6660ce8747884966a7eb0a57de7bfd0aafe17b0bacb89cac0757367"}
Jul 29 19:30:16.730253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.730202    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"611caf80820bcdca19c4074dd06163b490472e76c61e753e0504efef38a4fe23"}
Jul 29 19:30:16.731763 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.731727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9jgs2" event={"ID":"c2d113c2-5b6f-4091-bb1b-111b923d7dad","Type":"ContainerStarted","Data":"fa09360c1a32af0343943ccf29e9da7e8c9d774ddec53a4c6212c2824cebf21d"}
Jul 29 19:30:16.735620 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.735595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal" event={"ID":"ca69d5b906524dbafb4263c4750a6c45","Type":"ContainerStarted","Data":"c06178742c9028bad13eb94b1698fdc7fdb8b03a49cb60951cdb50fc22d7bdff"}
Jul 29 19:30:16.739620 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.739139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-crpff" event={"ID":"5fd2995c-1986-43db-b5ff-dd46e220a926","Type":"ContainerStarted","Data":"412f05389aba4235f52d457e8d1db4ed154739c6036fb52a9142e4d245e67ecb"}
Jul 29 19:30:16.740205 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.740184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h27gc" event={"ID":"1c633756-10c1-43d4-9dc0-b722ffa6deb0","Type":"ContainerStarted","Data":"62b83ef4799389116def2b084d10dec4531872ce57cec076b728f63331816c4c"}
Jul 29 19:30:16.741458 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.741438    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj" event={"ID":"af9419a1-7ae3-4842-86ef-ee9498a42799","Type":"ContainerStarted","Data":"12e1e227fc2461cb2b829e789f2182849d0a1f6f2e018f16b258487644698a42"}
Jul 29 19:30:16.742742 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.742721    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerStarted","Data":"9415671e4a05aef1e58c47ebbd525363ef193e725b7bcbf17387b54274ebafea"}
Jul 29 19:30:16.746834 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.746792    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6g6hg" podStartSLOduration=3.489726869 podStartE2EDuration="22.746779205s" totalImagesPullingTime="19.257052336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.745425945 +0000 UTC m=+22.703920854" watchObservedRunningTime="2026-07-29 19:30:16.746779205 +0000 UTC m=+22.705274098"
Jul 29 19:30:16.771290 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.771249    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9jgs2" podStartSLOduration=3.457720412 podStartE2EDuration="22.771234127s" totalImagesPullingTime="19.313513715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.771032076 +0000 UTC m=+22.729526969" watchObservedRunningTime="2026-07-29 19:30:16.771234127 +0000 UTC m=+22.729729020"
Jul 29 19:30:16.833739 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.833687    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hw7vt" podStartSLOduration=8.024988826 podStartE2EDuration="22.833673698s" totalImagesPullingTime="14.808684872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.833116561 +0000 UTC m=+22.791611455" watchObservedRunningTime="2026-07-29 19:30:16.833673698 +0000 UTC m=+22.792168591"
Jul 29 19:30:16.853998 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.853915    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-139.ec2.internal" podStartSLOduration=21.853893217 podStartE2EDuration="21.853893217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.853179514 +0000 UTC m=+22.811674409" watchObservedRunningTime="2026-07-29 19:30:16.853893217 +0000 UTC m=+22.812388112"
Jul 29 19:30:16.871713 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:16.871676    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h27gc" podStartSLOduration=8.074699127 podStartE2EDuration="22.871663079s" totalImagesPullingTime="14.796963952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.871557195 +0000 UTC m=+22.830052109" watchObservedRunningTime="2026-07-29 19:30:16.871663079 +0000 UTC m=+22.830157973"
Jul 29 19:30:17.745731 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.745651    2555 generic.go:356] "Generic (PLEG): container finished" podID="48b623d21ae99c3ed73e2aaafa3d4404" containerID="9120fb5e4809cf17fac01ad2749303ebfccd684d797e0655a88ea089532c26df" exitCode=0
Jul 29 19:30:17.746405 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.745739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal" event={"ID":"48b623d21ae99c3ed73e2aaafa3d4404","Type":"ContainerDied","Data":"9120fb5e4809cf17fac01ad2749303ebfccd684d797e0655a88ea089532c26df"}
Jul 29 19:30:17.747178 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.747147    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfb04e9c-60d5-47d2-af30-45bd0cf67021" containerID="9415671e4a05aef1e58c47ebbd525363ef193e725b7bcbf17387b54274ebafea" exitCode=0
Jul 29 19:30:17.747312 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.747222    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerDied","Data":"9415671e4a05aef1e58c47ebbd525363ef193e725b7bcbf17387b54274ebafea"}
Jul 29 19:30:17.750224 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.750207    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:30:17.750603 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.750561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"4bdbc4977f63c9fee94395cf1aa555888ff2c081a2836cf14cb2a5e180ce1684"}
Jul 29 19:30:17.751899 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.751869    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hmhxl" event={"ID":"2612c95d-e872-4e72-8fb6-cd7714290445","Type":"ContainerStarted","Data":"9009fe9c3ade2ae4a12d59f044f7e1d7145a22db483b95da63a62fdcc9d9a2bb"}
Jul 29 19:30:17.766834 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.766788    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-crpff" podStartSLOduration=4.230907015 podStartE2EDuration="23.766774928s" totalImagesPullingTime="19.535867913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.901483126 +0000 UTC m=+22.859978052" watchObservedRunningTime="2026-07-29 19:30:17.766774928 +0000 UTC m=+23.725269914"
Jul 29 19:30:17.896745 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:17.896711    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:30:18.039542 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.039386    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 19:30:18.530360 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.530253    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T19:30:18.039532747Z","UUID":"59b6cad1-4299-4061-b347-527a4b26b94e","Handler":null,"Name":"","Endpoint":""}
Jul 29 19:30:18.532119 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.532100    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 19:30:18.532258 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.532140    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 19:30:18.652742 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.652709    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:18.652907 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:18.652845    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:18.653191 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.653164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:18.653317 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:18.653233    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:18.653317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.653294    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:18.653437 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:18.653371    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:18.756637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.756598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal" event={"ID":"48b623d21ae99c3ed73e2aaafa3d4404","Type":"ContainerStarted","Data":"70cd08e832530a7f067528a05fafc21ee8e7f905393743b15a408d699156b036"}
Jul 29 19:30:18.759070 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.759041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj" event={"ID":"af9419a1-7ae3-4842-86ef-ee9498a42799","Type":"ContainerStarted","Data":"bdfbc93a13794d1cf86ba9f22d7e7e8925e9c3fc39ebf5dab8b267a2d7caa54a"}
Jul 29 19:30:18.784877 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.784797    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-139.ec2.internal" podStartSLOduration=23.784782082 podStartE2EDuration="23.784782082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:18.783986912 +0000 UTC m=+24.742481806" watchObservedRunningTime="2026-07-29 19:30:18.784782082 +0000 UTC m=+24.743276975"
Jul 29 19:30:18.785037 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:18.784878    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hmhxl" podStartSLOduration=9.946263784 podStartE2EDuration="24.78487338s" totalImagesPullingTime="14.838609596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:17.824186816 +0000 UTC m=+23.782681706" watchObservedRunningTime="2026-07-29 19:30:18.78487338 +0000 UTC m=+24.743368274"
Jul 29 19:30:19.763945 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:19.763910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj" event={"ID":"af9419a1-7ae3-4842-86ef-ee9498a42799","Type":"ContainerStarted","Data":"d14202b78609efe75c09835921b4c244ab76e87d153332e00c2b583e69f61540"}
Jul 29 19:30:19.767881 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:19.767858    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:30:19.768674 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:19.768638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"577f0b72eb1e9c1c7975419281666fcd2c63d7c7d39bae1b0c1918ec7782fdec"}
Jul 29 19:30:19.786268 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:19.786206    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dd9pj" podStartSLOduration=4.178927426 podStartE2EDuration="25.786193896s" totalImagesPullingTime="21.60726647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:19.78590974 +0000 UTC m=+25.744404624" watchObservedRunningTime="2026-07-29 19:30:19.786193896 +0000 UTC m=+25.744688787"
Jul 29 19:30:20.474219 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:20.474186    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:30:20.474937 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:20.474917    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:30:20.652665 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:20.652631    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:20.652844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:20.652674    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:20.652844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:20.652631    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:20.652844 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:20.652770    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:20.653009 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:20.652845    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:20.653009 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:20.652914    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:20.772183 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:20.772111    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h27gc"
Jul 29 19:30:22.653252 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.653089    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:22.653819 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.653110    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:22.653819 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:22.653325    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:22.653819 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.653110    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:22.653819 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:22.653447    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:22.653819 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:22.653462    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:22.776696 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.776665    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfb04e9c-60d5-47d2-af30-45bd0cf67021" containerID="4a3464be9023adfa9776b1a957c30dc4aaa08b0f04c7a1495635f7ae1e59f423" exitCode=0
Jul 29 19:30:22.776876 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.776754    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerDied","Data":"4a3464be9023adfa9776b1a957c30dc4aaa08b0f04c7a1495635f7ae1e59f423"}
Jul 29 19:30:22.780407 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.780388    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:30:22.780781 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.780761    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"5f4c6402818fc1aa70ea62469bcfe9696f2987df467a04d879894825dfab5344"}
Jul 29 19:30:22.783206 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:22.782226    2555 scope.go:117] "RemoveContainer" containerID="eaef15e0e6660ce8747884966a7eb0a57de7bfd0aafe17b0bacb89cac0757367"
Jul 29 19:30:23.788661 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.788635    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:30:23.789317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.789048    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" event={"ID":"fc12525f-9d1b-4f9b-b066-4781885c641d","Type":"ContainerStarted","Data":"39460d87018466b9683c86c76bd65b6fc7081a85868440b646f8e3a5efad2531"}
Jul 29 19:30:23.789317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.789287    2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 19:30:23.789567 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.789549    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:30:23.789681 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.789646    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:30:23.808749 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.808723    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:30:23.812013 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.811993    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:30:23.838424 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:23.838377    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rg625" podStartSLOduration=10.486397901 podStartE2EDuration="29.838360403s" totalImagesPullingTime="19.351962502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:23.837541464 +0000 UTC m=+29.796036379" watchObservedRunningTime="2026-07-29 19:30:23.838360403 +0000 UTC m=+29.796855358"
Jul 29 19:30:24.235287 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.235111    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g24sx"]
Jul 29 19:30:24.235431 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.235352    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:30:24.235470 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.235442    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:24.235545 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:24.235528    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:24.241597 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.241546    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mck7p"]
Jul 29 19:30:24.241730 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.241709    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:24.241840 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:24.241819    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:24.242236 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.242210    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gr9w2"]
Jul 29 19:30:24.242352 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.242305    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:24.242406 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:24.242383    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:24.792678 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.792645    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfb04e9c-60d5-47d2-af30-45bd0cf67021" containerID="d756157d357767188f7efcd8c6303ed73f6effd72b2f3db233d6961de1992812" exitCode=0
Jul 29 19:30:24.793070 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:24.792725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerDied","Data":"d756157d357767188f7efcd8c6303ed73f6effd72b2f3db233d6961de1992812"}
Jul 29 19:30:25.652752 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:25.652724    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:25.652918 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:25.652727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:25.652918 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:25.652735    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:25.653676 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:25.653273    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:25.653676 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:25.653287    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:25.653676 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:25.653617    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:26.800676 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:26.800634    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfb04e9c-60d5-47d2-af30-45bd0cf67021" containerID="dd663359ff97dfaf9ed49c897bb7c3cc60a320d22d89838b3273f73471a053ee" exitCode=0
Jul 29 19:30:26.800676 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:26.800672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerDied","Data":"dd663359ff97dfaf9ed49c897bb7c3cc60a320d22d89838b3273f73471a053ee"}
Jul 29 19:30:27.652838 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:27.652754    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:27.653012 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:27.652762    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:27.653012 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:27.652880    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:27.653012 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:27.652885    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gr9w2" podUID="e0d368bf-30b8-4d18-a17f-306eee453ddd"
Jul 29 19:30:27.653012 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:27.652947    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mck7p" podUID="8b11f949-b457-40d7-832a-6c054be24568"
Jul 29 19:30:27.653227 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:27.653036    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g24sx" podUID="1fb6effe-4bf8-46eb-b57c-414367796ec5"
Jul 29 19:30:28.328780 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:28.328748    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:28.328780 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:28.328791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:28.328837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.328945    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.328963    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.328995    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.329006    2555 projected.go:196] Error preparing data for projected volume kube-api-access-t95g4 for pod openshift-network-diagnostics/network-check-target-mck7p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.329005    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.329018    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs podName:1fb6effe-4bf8-46eb-b57c-414367796ec5 nodeName:}" failed. No retries permitted until 2026-07-29 19:31:00.329001638 +0000 UTC m=+66.287496513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs") pod "network-metrics-daemon-g24sx" (UID: "1fb6effe-4bf8-46eb-b57c-414367796ec5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.329047    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4 podName:8b11f949-b457-40d7-832a-6c054be24568 nodeName:}" failed. No retries permitted until 2026-07-29 19:31:00.329035369 +0000 UTC m=+66.287530245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t95g4" (UniqueName: "kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4") pod "network-check-target-mck7p" (UID: "8b11f949-b457-40d7-832a-6c054be24568") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 19:30:28.329289 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:28.329061    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret podName:e0d368bf-30b8-4d18-a17f-306eee453ddd nodeName:}" failed. No retries permitted until 2026-07-29 19:31:00.329054383 +0000 UTC m=+66.287549254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret") pod "global-pull-secret-syncer-gr9w2" (UID: "e0d368bf-30b8-4d18-a17f-306eee453ddd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 19:30:29.399332 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.399288    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-139.ec2.internal" event="NodeReady"
Jul 29 19:30:29.399901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.399459    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 19:30:29.467611 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.467564    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cddmj"]
Jul 29 19:30:29.482464 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.482426    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wzmhh"]
Jul 29 19:30:29.482663 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.482569    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.484625 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.484603    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 19:30:29.484744 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.484628    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8z4sg\""
Jul 29 19:30:29.484899 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.484881    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 19:30:29.492897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.492704    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wzmhh"]
Jul 29 19:30:29.492897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.492739    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cddmj"]
Jul 29 19:30:29.492897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.492831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.494953 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.494938    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ck8zl\""
Jul 29 19:30:29.495116 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.494982    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 19:30:29.495211 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.494983    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 19:30:29.495211 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.495020    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 19:30:29.638472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.638436    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8838db34-5e79-4ad7-954d-956d6d35ed2f-tmp-dir\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.638736 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.638498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29mmv\" (UniqueName: \"kubernetes.io/projected/8838db34-5e79-4ad7-954d-956d6d35ed2f-kube-api-access-29mmv\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.638736 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.638521    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd75f887-a524-4013-a82a-6d959ed55c52-cert\") pod \"ingress-canary-wzmhh\" (UID: \"cd75f887-a524-4013-a82a-6d959ed55c52\") " pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.638736 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.638547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8838db34-5e79-4ad7-954d-956d6d35ed2f-config-volume\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.638736 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.638593    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwb9x\" (UniqueName: \"kubernetes.io/projected/cd75f887-a524-4013-a82a-6d959ed55c52-kube-api-access-jwb9x\") pod \"ingress-canary-wzmhh\" (UID: \"cd75f887-a524-4013-a82a-6d959ed55c52\") " pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.638736 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.638680    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.652666 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.652593    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:30:29.652810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.652718    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:30:29.652810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.652734    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:30:29.659472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.659445    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:30:29.659472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.659471    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:30:29.659692 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.659522    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:30:29.659692 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.659535    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xr6nn\""
Jul 29 19:30:29.659692 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.659452    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qwcp5\""
Jul 29 19:30:29.664524 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.664502    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:30:29.739225 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.739421 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739247    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8838db34-5e79-4ad7-954d-956d6d35ed2f-tmp-dir\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.739421 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:29.739405    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:30:29.739537 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739454    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mmv\" (UniqueName: \"kubernetes.io/projected/8838db34-5e79-4ad7-954d-956d6d35ed2f-kube-api-access-29mmv\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.739537 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:29.739491    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls podName:8838db34-5e79-4ad7-954d-956d6d35ed2f nodeName:}" failed. No retries permitted until 2026-07-29 19:30:30.239469387 +0000 UTC m=+36.197964266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls") pod "dns-default-cddmj" (UID: "8838db34-5e79-4ad7-954d-956d6d35ed2f") : secret "dns-default-metrics-tls" not found
Jul 29 19:30:29.739681 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739536    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd75f887-a524-4013-a82a-6d959ed55c52-cert\") pod \"ingress-canary-wzmhh\" (UID: \"cd75f887-a524-4013-a82a-6d959ed55c52\") " pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.739681 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739565    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8838db34-5e79-4ad7-954d-956d6d35ed2f-config-volume\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.739681 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwb9x\" (UniqueName: \"kubernetes.io/projected/cd75f887-a524-4013-a82a-6d959ed55c52-kube-api-access-jwb9x\") pod \"ingress-canary-wzmhh\" (UID: \"cd75f887-a524-4013-a82a-6d959ed55c52\") " pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.739681 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.739623    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8838db34-5e79-4ad7-954d-956d6d35ed2f-tmp-dir\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.740260 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.740240    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8838db34-5e79-4ad7-954d-956d6d35ed2f-config-volume\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.745172 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.745147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd75f887-a524-4013-a82a-6d959ed55c52-cert\") pod \"ingress-canary-wzmhh\" (UID: \"cd75f887-a524-4013-a82a-6d959ed55c52\") " pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.754340 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.754313    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mmv\" (UniqueName: \"kubernetes.io/projected/8838db34-5e79-4ad7-954d-956d6d35ed2f-kube-api-access-29mmv\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:29.757547 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.757523    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwb9x\" (UniqueName: \"kubernetes.io/projected/cd75f887-a524-4013-a82a-6d959ed55c52-kube-api-access-jwb9x\") pod \"ingress-canary-wzmhh\" (UID: \"cd75f887-a524-4013-a82a-6d959ed55c52\") " pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.803498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.803464    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wzmhh"
Jul 29 19:30:29.940514 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.940483    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"]
Jul 29 19:30:29.952224 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.952200    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"
Jul 29 19:30:29.954244 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.954220    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v582r\""
Jul 29 19:30:29.954438 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.954421    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 19:30:29.954500 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.954465    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 19:30:29.960392 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:29.960369    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"]
Jul 29 19:30:30.041177 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.041139    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvfh\" (UniqueName: \"kubernetes.io/projected/b8392df1-96e0-490d-b551-54fef247c2fd-kube-api-access-dlvfh\") pod \"migrator-6865c9967-spp7q\" (UID: \"b8392df1-96e0-490d-b551-54fef247c2fd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"
Jul 29 19:30:30.141543 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.141500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvfh\" (UniqueName: \"kubernetes.io/projected/b8392df1-96e0-490d-b551-54fef247c2fd-kube-api-access-dlvfh\") pod \"migrator-6865c9967-spp7q\" (UID: \"b8392df1-96e0-490d-b551-54fef247c2fd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"
Jul 29 19:30:30.160348 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.160314    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dlkwn"]
Jul 29 19:30:30.167112 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.167089    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvfh\" (UniqueName: \"kubernetes.io/projected/b8392df1-96e0-490d-b551-54fef247c2fd-kube-api-access-dlvfh\") pod \"migrator-6865c9967-spp7q\" (UID: \"b8392df1-96e0-490d-b551-54fef247c2fd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"
Jul 29 19:30:30.184268 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.183652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlkwn"]
Jul 29 19:30:30.184268 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.183794    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.185900 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.185883    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 19:30:30.186014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.185907    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 19:30:30.186137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.186117    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 19:30:30.186358 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.186340    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 19:30:30.186358 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.186354    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8twzb\""
Jul 29 19:30:30.242242 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.242212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:30.242443 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.242378    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:30:30.242443 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.242437    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls podName:8838db34-5e79-4ad7-954d-956d6d35ed2f nodeName:}" failed. No retries permitted until 2026-07-29 19:30:31.242424718 +0000 UTC m=+37.200919590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls") pod "dns-default-cddmj" (UID: "8838db34-5e79-4ad7-954d-956d6d35ed2f") : secret "dns-default-metrics-tls" not found
Jul 29 19:30:30.265350 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.265319    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"
Jul 29 19:30:30.343137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.343094    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/639a026f-f76d-4c50-ad25-20f56fe83839-crio-socket\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.343352 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.343148    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/639a026f-f76d-4c50-ad25-20f56fe83839-data-volume\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.343352 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.343208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.343352 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.343303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.343352 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.343342    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpjf\" (UniqueName: \"kubernetes.io/projected/639a026f-f76d-4c50-ad25-20f56fe83839-kube-api-access-srpjf\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444037 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.443949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444037 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.444018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpjf\" (UniqueName: \"kubernetes.io/projected/639a026f-f76d-4c50-ad25-20f56fe83839-kube-api-access-srpjf\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.444063    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/639a026f-f76d-4c50-ad25-20f56fe83839-crio-socket\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.444082    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.444101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/639a026f-f76d-4c50-ad25-20f56fe83839-data-volume\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.444164    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:30.944142874 +0000 UTC m=+36.902637745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : configmap "kube-rbac-proxy" not found
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.444234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.444258    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/639a026f-f76d-4c50-ad25-20f56fe83839-crio-socket\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.444353    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.444397    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:30.944382678 +0000 UTC m=+36.902877551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:30.444610 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.444438    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/639a026f-f76d-4c50-ad25-20f56fe83839-data-volume\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.456289 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.456264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpjf\" (UniqueName: \"kubernetes.io/projected/639a026f-f76d-4c50-ad25-20f56fe83839-kube-api-access-srpjf\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.948488 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.948272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.948700 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.948399    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:30:30.948700 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.948605    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:30.948700 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.948646    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:31.948616865 +0000 UTC m=+37.907111738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : configmap "kube-rbac-proxy" not found
Jul 29 19:30:30.948870 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.948705    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:30.948870 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:30.948753    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:31.948738368 +0000 UTC m=+37.907233241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:30.968472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.968445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wzmhh"]
Jul 29 19:30:30.973670 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:30.973641    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q"]
Jul 29 19:30:31.024625 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:31.024569    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd75f887_a524_4013_a82a_6d959ed55c52.slice/crio-1ba358d0089b726cac1d88a80677a6f40933e476c19e8308b6b1a5a40f76e27e WatchSource:0}: Error finding container 1ba358d0089b726cac1d88a80677a6f40933e476c19e8308b6b1a5a40f76e27e: Status 404 returned error can't find the container with id 1ba358d0089b726cac1d88a80677a6f40933e476c19e8308b6b1a5a40f76e27e
Jul 29 19:30:31.025307 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:31.025194    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8392df1_96e0_490d_b551_54fef247c2fd.slice/crio-89883b6a1054abd052e7faf7faf73dcaa27447cd6a3eb4e9fe48967ce01a4636 WatchSource:0}: Error finding container 89883b6a1054abd052e7faf7faf73dcaa27447cd6a3eb4e9fe48967ce01a4636: Status 404 returned error can't find the container with id 89883b6a1054abd052e7faf7faf73dcaa27447cd6a3eb4e9fe48967ce01a4636
Jul 29 19:30:31.252534 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.252506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:31.252667 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:31.252647    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:30:31.252713 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:31.252705    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls podName:8838db34-5e79-4ad7-954d-956d6d35ed2f nodeName:}" failed. No retries permitted until 2026-07-29 19:30:33.252691667 +0000 UTC m=+39.211186540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls") pod "dns-default-cddmj" (UID: "8838db34-5e79-4ad7-954d-956d6d35ed2f") : secret "dns-default-metrics-tls" not found
Jul 29 19:30:31.815299 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.815264    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wzmhh" event={"ID":"cd75f887-a524-4013-a82a-6d959ed55c52","Type":"ContainerStarted","Data":"1ba358d0089b726cac1d88a80677a6f40933e476c19e8308b6b1a5a40f76e27e"}
Jul 29 19:30:31.816757 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.816723    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q" event={"ID":"b8392df1-96e0-490d-b551-54fef247c2fd","Type":"ContainerStarted","Data":"89883b6a1054abd052e7faf7faf73dcaa27447cd6a3eb4e9fe48967ce01a4636"}
Jul 29 19:30:31.820897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.820402    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfb04e9c-60d5-47d2-af30-45bd0cf67021" containerID="c0fcc30ad418c1cac58a6d4fb3dc5356138eb07d7e8f160faa87378b0e1bbd95" exitCode=0
Jul 29 19:30:31.820897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.820437    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerDied","Data":"c0fcc30ad418c1cac58a6d4fb3dc5356138eb07d7e8f160faa87378b0e1bbd95"}
Jul 29 19:30:31.957440 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.957267    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:31.957440 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:31.957381    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:31.961286 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:31.958991    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:31.961286 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:31.959065    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:33.959038699 +0000 UTC m=+39.917533580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:31.961286 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:31.958994    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:30:31.961286 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:31.959284    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:33.959264914 +0000 UTC m=+39.917759794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : configmap "kube-rbac-proxy" not found
Jul 29 19:30:32.199058 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.198033    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-h55l9"]
Jul 29 19:30:32.201280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.200656    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.203341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.202897    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 29 19:30:32.203341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.203099    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 29 19:30:32.203341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.203133    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lrf6x\""
Jul 29 19:30:32.203341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.203181    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 29 19:30:32.203341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.203138    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 29 19:30:32.210887 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.210823    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-h55l9"]
Jul 29 19:30:32.361470 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.361438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-signing-cabundle\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.361470 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.361472    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrghz\" (UniqueName: \"kubernetes.io/projected/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-kube-api-access-nrghz\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.361726 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.361510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-signing-key\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.461898 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.461874    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-signing-cabundle\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.462028 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.461910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrghz\" (UniqueName: \"kubernetes.io/projected/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-kube-api-access-nrghz\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.462028 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.461968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-signing-key\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.462629 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.462607    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-signing-cabundle\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.465657 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.465491    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-signing-key\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.471730 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.471431    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrghz\" (UniqueName: \"kubernetes.io/projected/a362a0ad-cf4e-447e-a6e5-2c35d633d5fe-kube-api-access-nrghz\") pod \"service-ca-658d7b7c78-h55l9\" (UID: \"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe\") " pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.511573 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.511526    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-h55l9"
Jul 29 19:30:32.658499 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.658426    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-h55l9"]
Jul 29 19:30:32.756791 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.756694    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6g6hg_16d4adba-8f03-4c88-96d3-cfc9449182f7/dns-node-resolver/0.log"
Jul 29 19:30:32.825659 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.825618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q" event={"ID":"b8392df1-96e0-490d-b551-54fef247c2fd","Type":"ContainerStarted","Data":"ec4a5860c31428e08d2808f2da8fae21c4881a15a5b08a79535c1c667e02991b"}
Jul 29 19:30:32.826062 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.825686    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q" event={"ID":"b8392df1-96e0-490d-b551-54fef247c2fd","Type":"ContainerStarted","Data":"c79dc03ead84e76f0d9ac2750588a4d9e70f5942009fe3bafac5cab9eddbf841"}
Jul 29 19:30:32.828995 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.828968    2555 generic.go:356] "Generic (PLEG): container finished" podID="bfb04e9c-60d5-47d2-af30-45bd0cf67021" containerID="2d35f073f7d7dcd8c99c798464c694c6f2723bdbc6b98235fbbbfda282ead27e" exitCode=0
Jul 29 19:30:32.829155 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.829003    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerDied","Data":"2d35f073f7d7dcd8c99c798464c694c6f2723bdbc6b98235fbbbfda282ead27e"}
Jul 29 19:30:32.849665 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:32.849615    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-spp7q" podStartSLOduration=2.443388099 podStartE2EDuration="3.849599145s" totalImagesPullingTime="1.406211046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:32.848909408 +0000 UTC m=+38.807404316" watchObservedRunningTime="2026-07-29 19:30:32.849599145 +0000 UTC m=+38.808094036"
Jul 29 19:30:33.267957 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.267920    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:33.268148 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:33.268072    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:30:33.268220 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:33.268151    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls podName:8838db34-5e79-4ad7-954d-956d6d35ed2f nodeName:}" failed. No retries permitted until 2026-07-29 19:30:37.268131038 +0000 UTC m=+43.226625916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls") pod "dns-default-cddmj" (UID: "8838db34-5e79-4ad7-954d-956d6d35ed2f") : secret "dns-default-metrics-tls" not found
Jul 29 19:30:33.281131 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:33.281105    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda362a0ad_cf4e_447e_a6e5_2c35d633d5fe.slice/crio-8d56e5db877094ff3dc1a0db4d849910b32e62c17a424ca8840073fa6128657a WatchSource:0}: Error finding container 8d56e5db877094ff3dc1a0db4d849910b32e62c17a424ca8840073fa6128657a: Status 404 returned error can't find the container with id 8d56e5db877094ff3dc1a0db4d849910b32e62c17a424ca8840073fa6128657a
Jul 29 19:30:33.761275 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.761102    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hw7vt_0c822540-948e-4f1e-8189-00f4c8bff2b8/node-ca/0.log"
Jul 29 19:30:33.836739 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.836705    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qp955" event={"ID":"bfb04e9c-60d5-47d2-af30-45bd0cf67021","Type":"ContainerStarted","Data":"f1d69069eefa36564eb539d17b2dc3e9d475bf90743613f10c51e9b8b5e3d45f"}
Jul 29 19:30:33.838566 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.838540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wzmhh" event={"ID":"cd75f887-a524-4013-a82a-6d959ed55c52","Type":"ContainerStarted","Data":"f2b93d932cf5ce497aec19393c3864483a262829ec3331e8ef160c452e206e85"}
Jul 29 19:30:33.841173 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.841004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-h55l9" event={"ID":"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe","Type":"ContainerStarted","Data":"8d56e5db877094ff3dc1a0db4d849910b32e62c17a424ca8840073fa6128657a"}
Jul 29 19:30:33.867733 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.867669    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qp955" podStartSLOduration=10.052777326 podStartE2EDuration="39.867648682s" totalImagesPullingTime="29.814871356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:33.866371393 +0000 UTC m=+39.824866301" watchObservedRunningTime="2026-07-29 19:30:33.867648682 +0000 UTC m=+39.826143577"
Jul 29 19:30:33.886030 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.885971    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wzmhh" podStartSLOduration=2.6018162240000002 podStartE2EDuration="4.88595208s" totalImagesPullingTime="2.284135856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:33.885509883 +0000 UTC m=+39.844004802" watchObservedRunningTime="2026-07-29 19:30:33.88595208 +0000 UTC m=+39.844446974"
Jul 29 19:30:33.972886 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.972844    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:33.973085 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:33.973033    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:33.973160 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:33.973101    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:37.973082489 +0000 UTC m=+43.931577375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:33.973160 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:33.973145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:33.973434 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:33.973413    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:30:33.973515 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:33.973463    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:37.973448945 +0000 UTC m=+43.931943817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : configmap "kube-rbac-proxy" not found
Jul 29 19:30:34.957332 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:34.957297    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wzmhh_cd75f887-a524-4013-a82a-6d959ed55c52/serve-healthcheck-canary/0.log"
Jul 29 19:30:35.157202 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:35.157173    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-spp7q_b8392df1-96e0-490d-b551-54fef247c2fd/migrator/0.log"
Jul 29 19:30:35.356557 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:35.356479    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-spp7q_b8392df1-96e0-490d-b551-54fef247c2fd/graceful-termination/0.log"
Jul 29 19:30:35.847210 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:35.847176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-h55l9" event={"ID":"a362a0ad-cf4e-447e-a6e5-2c35d633d5fe","Type":"ContainerStarted","Data":"451110937c4f92afd6ac53294a904db97f3967972ee94aeb6751d2af2962c71b"}
Jul 29 19:30:35.872356 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:35.872309    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-h55l9" podStartSLOduration=2.123080423 podStartE2EDuration="3.872291985s" totalImagesPullingTime="1.749211562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:35.871681508 +0000 UTC m=+41.830176421" watchObservedRunningTime="2026-07-29 19:30:35.872291985 +0000 UTC m=+41.830786879"
Jul 29 19:30:37.299677 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:37.299640    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:37.300141 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:37.299822    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 19:30:37.300141 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:37.299897    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls podName:8838db34-5e79-4ad7-954d-956d6d35ed2f nodeName:}" failed. No retries permitted until 2026-07-29 19:30:45.299876313 +0000 UTC m=+51.258371189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls") pod "dns-default-cddmj" (UID: "8838db34-5e79-4ad7-954d-956d6d35ed2f") : secret "dns-default-metrics-tls" not found
Jul 29 19:30:38.005728 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:38.005695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:38.005905 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:38.005757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:38.005905 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:38.005863    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:38.005970 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:38.005936    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:46.005920536 +0000 UTC m=+51.964415409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:38.005970 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:38.005863    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 19:30:38.006038 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:38.006000    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:30:46.005987449 +0000 UTC m=+51.964482321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : configmap "kube-rbac-proxy" not found
Jul 29 19:30:45.360141 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:45.360099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:45.364014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:45.363979    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8838db34-5e79-4ad7-954d-956d6d35ed2f-metrics-tls\") pod \"dns-default-cddmj\" (UID: \"8838db34-5e79-4ad7-954d-956d6d35ed2f\") " pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:45.396084 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:45.396054    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:45.570360 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:45.570330    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cddmj"]
Jul 29 19:30:45.573694 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:45.573668    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8838db34_5e79_4ad7_954d_956d6d35ed2f.slice/crio-399d66ec6b0ecd1d7b3e9d1d7126c315f27286936cbf78718758f9b1229d0cfe WatchSource:0}: Error finding container 399d66ec6b0ecd1d7b3e9d1d7126c315f27286936cbf78718758f9b1229d0cfe: Status 404 returned error can't find the container with id 399d66ec6b0ecd1d7b3e9d1d7126c315f27286936cbf78718758f9b1229d0cfe
Jul 29 19:30:45.872984 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:45.872943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cddmj" event={"ID":"8838db34-5e79-4ad7-954d-956d6d35ed2f","Type":"ContainerStarted","Data":"399d66ec6b0ecd1d7b3e9d1d7126c315f27286936cbf78718758f9b1229d0cfe"}
Jul 29 19:30:46.065860 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:46.065830    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:46.066022 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:46.065898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:46.066022 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:46.066008    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:46.066088 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:30:46.066071    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls podName:639a026f-f76d-4c50-ad25-20f56fe83839 nodeName:}" failed. No retries permitted until 2026-07-29 19:31:02.066053845 +0000 UTC m=+68.024548723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dlkwn" (UID: "639a026f-f76d-4c50-ad25-20f56fe83839") : secret "insights-runtime-extractor-tls" not found
Jul 29 19:30:46.066335 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:46.066315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/639a026f-f76d-4c50-ad25-20f56fe83839-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:30:47.880398 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:47.880367    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cddmj" event={"ID":"8838db34-5e79-4ad7-954d-956d6d35ed2f","Type":"ContainerStarted","Data":"689a982be5a8fc790102488084f675a4367f360d2346adf89ecaa63028c8d2da"}
Jul 29 19:30:48.884480 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:48.884441    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cddmj" event={"ID":"8838db34-5e79-4ad7-954d-956d6d35ed2f","Type":"ContainerStarted","Data":"6cb8569cba0062994005b2d1fc948aa35957a3654248090c45cf28872e3af96d"}
Jul 29 19:30:48.884896 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:48.884601    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:48.903224 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:48.903175    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cddmj" podStartSLOduration=17.746255231 podStartE2EDuration="19.903161616s" totalImagesPullingTime="2.156906385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:48.902363441 +0000 UTC m=+54.860858349" watchObservedRunningTime="2026-07-29 19:30:48.903161616 +0000 UTC m=+54.861656501"
Jul 29 19:30:55.349861 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.349823    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"]
Jul 29 19:30:55.382501 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.382462    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"]
Jul 29 19:30:55.382685 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.382602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.385266 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.385244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 19:30:55.385828 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.385808    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 19:30:55.385958 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.385860    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 19:30:55.386086 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.386064    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 19:30:55.427228 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.427192    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4b31063-bda6-49f2-8c39-ab5a5b516391-tmp\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.427384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.427257    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vb6d\" (UniqueName: \"kubernetes.io/projected/b4b31063-bda6-49f2-8c39-ab5a5b516391-kube-api-access-8vb6d\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.427384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.427314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b4b31063-bda6-49f2-8c39-ab5a5b516391-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.448956 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.448929    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-xnzj9"]
Jul 29 19:30:55.467083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.467056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xnzj9"]
Jul 29 19:30:55.467215 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.467160    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:30:55.469205 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.469184    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 19:30:55.469335 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.469185    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cmhfp\""
Jul 29 19:30:55.469335 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.469307    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 19:30:55.528478 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.528448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4b31063-bda6-49f2-8c39-ab5a5b516391-tmp\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.528637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.528493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vb6d\" (UniqueName: \"kubernetes.io/projected/b4b31063-bda6-49f2-8c39-ab5a5b516391-kube-api-access-8vb6d\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.528637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.528550    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87985\" (UniqueName: \"kubernetes.io/projected/26f89cc3-3bde-4629-b636-f83c62f29c3f-kube-api-access-87985\") pod \"downloads-7c97dc4555-xnzj9\" (UID: \"26f89cc3-3bde-4629-b636-f83c62f29c3f\") " pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:30:55.528637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.528572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b4b31063-bda6-49f2-8c39-ab5a5b516391-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.528853 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.528833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4b31063-bda6-49f2-8c39-ab5a5b516391-tmp\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.531052 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.531034    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b4b31063-bda6-49f2-8c39-ab5a5b516391-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.549006 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.548973    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vb6d\" (UniqueName: \"kubernetes.io/projected/b4b31063-bda6-49f2-8c39-ab5a5b516391-kube-api-access-8vb6d\") pod \"klusterlet-addon-workmgr-5fd7b6f779-b4666\" (UID: \"b4b31063-bda6-49f2-8c39-ab5a5b516391\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.554436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.554410    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-577699577-shb6z"]
Jul 29 19:30:55.572416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.572393    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.574548 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.574529    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 19:30:55.574665 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.574530    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 19:30:55.574791 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.574773    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 19:30:55.574862 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.574808    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-95c74\""
Jul 29 19:30:55.577905 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.577885    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577699577-shb6z"]
Jul 29 19:30:55.580002 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.579979    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 19:30:55.629118 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629048    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9aec82e6-db34-4e43-8bb4-8189e02c2a06-installation-pull-secrets\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629118 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9aec82e6-db34-4e43-8bb4-8189e02c2a06-trusted-ca\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629305 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-registry-tls\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629305 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629220    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87985\" (UniqueName: \"kubernetes.io/projected/26f89cc3-3bde-4629-b636-f83c62f29c3f-kube-api-access-87985\") pod \"downloads-7c97dc4555-xnzj9\" (UID: \"26f89cc3-3bde-4629-b636-f83c62f29c3f\") " pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:30:55.629305 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629240    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9aec82e6-db34-4e43-8bb4-8189e02c2a06-registry-certificates\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629305 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9aec82e6-db34-4e43-8bb4-8189e02c2a06-ca-trust-extracted\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629305 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9aec82e6-db34-4e43-8bb4-8189e02c2a06-image-registry-private-configuration\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629544 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629324    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-bound-sa-token\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.629544 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.629384    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpvl\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-kube-api-access-frpvl\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.638286 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.638247    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87985\" (UniqueName: \"kubernetes.io/projected/26f89cc3-3bde-4629-b636-f83c62f29c3f-kube-api-access-87985\") pod \"downloads-7c97dc4555-xnzj9\" (UID: \"26f89cc3-3bde-4629-b636-f83c62f29c3f\") " pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:30:55.693541 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.693509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:30:55.729961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.729934    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9aec82e6-db34-4e43-8bb4-8189e02c2a06-image-registry-private-configuration\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730086 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.729976    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-bound-sa-token\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730086 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730015    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpvl\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-kube-api-access-frpvl\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730086 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9aec82e6-db34-4e43-8bb4-8189e02c2a06-installation-pull-secrets\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730252 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730103    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9aec82e6-db34-4e43-8bb4-8189e02c2a06-trusted-ca\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730311 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-registry-tls\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730413 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9aec82e6-db34-4e43-8bb4-8189e02c2a06-registry-certificates\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730477 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9aec82e6-db34-4e43-8bb4-8189e02c2a06-ca-trust-extracted\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.730878 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.730837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9aec82e6-db34-4e43-8bb4-8189e02c2a06-ca-trust-extracted\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.731079 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.731020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9aec82e6-db34-4e43-8bb4-8189e02c2a06-trusted-ca\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.731403 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.731381    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9aec82e6-db34-4e43-8bb4-8189e02c2a06-registry-certificates\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.732710 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.732688    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9aec82e6-db34-4e43-8bb4-8189e02c2a06-image-registry-private-configuration\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.733191 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.733170    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-registry-tls\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.733262 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.733191    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9aec82e6-db34-4e43-8bb4-8189e02c2a06-installation-pull-secrets\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.748006 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.747971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-bound-sa-token\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.748103 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.748020    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpvl\" (UniqueName: \"kubernetes.io/projected/9aec82e6-db34-4e43-8bb4-8189e02c2a06-kube-api-access-frpvl\") pod \"image-registry-577699577-shb6z\" (UID: \"9aec82e6-db34-4e43-8bb4-8189e02c2a06\") " pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.775839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.775804    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:30:55.810155 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.809954    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rg625"
Jul 29 19:30:55.855818 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.855472    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"]
Jul 29 19:30:55.859489 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:55.859462    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b31063_bda6_49f2_8c39_ab5a5b516391.slice/crio-dfab4db864062349a864f149e932e768fa765b65b55c00d082d59878d0902441 WatchSource:0}: Error finding container dfab4db864062349a864f149e932e768fa765b65b55c00d082d59878d0902441: Status 404 returned error can't find the container with id dfab4db864062349a864f149e932e768fa765b65b55c00d082d59878d0902441
Jul 29 19:30:55.882428 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.882359    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:55.907478 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.907439    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666" event={"ID":"b4b31063-bda6-49f2-8c39-ab5a5b516391","Type":"ContainerStarted","Data":"dfab4db864062349a864f149e932e768fa765b65b55c00d082d59878d0902441"}
Jul 29 19:30:55.908377 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:55.908350    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xnzj9"]
Jul 29 19:30:55.912774 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:55.912746    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f89cc3_3bde_4629_b636_f83c62f29c3f.slice/crio-64eeef3f0a92596a66f0cb9fdc7dcce37ca63e1b1a93f02e59f13440fd837d5e WatchSource:0}: Error finding container 64eeef3f0a92596a66f0cb9fdc7dcce37ca63e1b1a93f02e59f13440fd837d5e: Status 404 returned error can't find the container with id 64eeef3f0a92596a66f0cb9fdc7dcce37ca63e1b1a93f02e59f13440fd837d5e
Jul 29 19:30:56.023482 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:56.023446    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577699577-shb6z"]
Jul 29 19:30:56.027313 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:30:56.027291    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aec82e6_db34_4e43_8bb4_8189e02c2a06.slice/crio-1a43cd1e8c06c571fab4d63831ec4b0af126f95ec6d0d0a8211e44eba1064424 WatchSource:0}: Error finding container 1a43cd1e8c06c571fab4d63831ec4b0af126f95ec6d0d0a8211e44eba1064424: Status 404 returned error can't find the container with id 1a43cd1e8c06c571fab4d63831ec4b0af126f95ec6d0d0a8211e44eba1064424
Jul 29 19:30:56.912301 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:56.912261    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xnzj9" event={"ID":"26f89cc3-3bde-4629-b636-f83c62f29c3f","Type":"ContainerStarted","Data":"64eeef3f0a92596a66f0cb9fdc7dcce37ca63e1b1a93f02e59f13440fd837d5e"}
Jul 29 19:30:56.915024 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:56.914121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577699577-shb6z" event={"ID":"9aec82e6-db34-4e43-8bb4-8189e02c2a06","Type":"ContainerStarted","Data":"3a3d0127f4e7cd1ae028e3f8a6165c0dc6c109523d4f41b50898977ac1f348ec"}
Jul 29 19:30:56.915024 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:56.914172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577699577-shb6z" event={"ID":"9aec82e6-db34-4e43-8bb4-8189e02c2a06","Type":"ContainerStarted","Data":"1a43cd1e8c06c571fab4d63831ec4b0af126f95ec6d0d0a8211e44eba1064424"}
Jul 29 19:30:56.915024 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:56.914924    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:30:56.934515 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:56.933770    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-577699577-shb6z" podStartSLOduration=1.93375228 podStartE2EDuration="1.93375228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:56.933237151 +0000 UTC m=+62.891732056" watchObservedRunningTime="2026-07-29 19:30:56.93375228 +0000 UTC m=+62.892247176"
Jul 29 19:30:58.890425 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:58.890392    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cddmj"
Jul 29 19:30:59.943475 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.943440    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"]
Jul 29 19:30:59.946985 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.946330    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:30:59.948702 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.948680    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 19:30:59.948820 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.948753    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 19:30:59.949452 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.949418    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 19:30:59.951195 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.949647    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6nj66\""
Jul 29 19:30:59.951195 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.949878    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 19:30:59.951195 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.950077    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 19:30:59.954520 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.954500    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"]
Jul 29 19:30:59.970163 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.970134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e2627-48d0-4286-a849-2d9c667293cc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:30:59.970267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.970195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8nj\" (UniqueName: \"kubernetes.io/projected/3b6e2627-48d0-4286-a849-2d9c667293cc-kube-api-access-9x8nj\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:30:59.970322 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.970286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b6e2627-48d0-4286-a849-2d9c667293cc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:30:59.970370 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:30:59.970318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b6e2627-48d0-4286-a849-2d9c667293cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.070923 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.070889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8nj\" (UniqueName: \"kubernetes.io/projected/3b6e2627-48d0-4286-a849-2d9c667293cc-kube-api-access-9x8nj\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.071100 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.071000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b6e2627-48d0-4286-a849-2d9c667293cc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.071100 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.071028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b6e2627-48d0-4286-a849-2d9c667293cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.071100 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.071062    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e2627-48d0-4286-a849-2d9c667293cc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.071935 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.071906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e2627-48d0-4286-a849-2d9c667293cc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.074150 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.074122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b6e2627-48d0-4286-a849-2d9c667293cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.074233 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.074180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b6e2627-48d0-4286-a849-2d9c667293cc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.081014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.080977    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8nj\" (UniqueName: \"kubernetes.io/projected/3b6e2627-48d0-4286-a849-2d9c667293cc-kube-api-access-9x8nj\") pod \"prometheus-operator-84d69f794d-4zlg5\" (UID: \"3b6e2627-48d0-4286-a849-2d9c667293cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.262091 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.262012    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"
Jul 29 19:31:00.373948 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.373908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:31:00.374152 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.374010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:31:00.374152 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.374063    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:31:00.376284 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.376254    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 19:31:00.376527 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.376508    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 19:31:00.376879 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.376725    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 19:31:00.386953 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.386929    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 19:31:00.387995 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.387972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fb6effe-4bf8-46eb-b57c-414367796ec5-metrics-certs\") pod \"network-metrics-daemon-g24sx\" (UID: \"1fb6effe-4bf8-46eb-b57c-414367796ec5\") " pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:31:00.395177 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.395150    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0d368bf-30b8-4d18-a17f-306eee453ddd-original-pull-secret\") pod \"global-pull-secret-syncer-gr9w2\" (UID: \"e0d368bf-30b8-4d18-a17f-306eee453ddd\") " pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:31:00.397941 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.397891    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t95g4\" (UniqueName: \"kubernetes.io/projected/8b11f949-b457-40d7-832a-6c054be24568-kube-api-access-t95g4\") pod \"network-check-target-mck7p\" (UID: \"8b11f949-b457-40d7-832a-6c054be24568\") " pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:31:00.567050 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.566966    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qwcp5\""
Jul 29 19:31:00.571630 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.571598    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gr9w2"
Jul 29 19:31:00.575469 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.575444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:31:00.578964 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.578793    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xr6nn\""
Jul 29 19:31:00.587356 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:00.587334    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g24sx"
Jul 29 19:31:01.128960 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.128931    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g24sx"]
Jul 29 19:31:01.135547 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:01.133351    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb6effe_4bf8_46eb_b57c_414367796ec5.slice/crio-49b8dcf8411ecc95ad886be91e51e9faada739e68b4075208e76a973b80f4e31 WatchSource:0}: Error finding container 49b8dcf8411ecc95ad886be91e51e9faada739e68b4075208e76a973b80f4e31: Status 404 returned error can't find the container with id 49b8dcf8411ecc95ad886be91e51e9faada739e68b4075208e76a973b80f4e31
Jul 29 19:31:01.358943 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.358852    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gr9w2"]
Jul 29 19:31:01.363213 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.363183    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mck7p"]
Jul 29 19:31:01.367455 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.367351    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-4zlg5"]
Jul 29 19:31:01.367945 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:01.367889    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b11f949_b457_40d7_832a_6c054be24568.slice/crio-db37b952e28997d5bf70eba2d6bdb5ca840aa46529e579c0306df2763868b2e9 WatchSource:0}: Error finding container db37b952e28997d5bf70eba2d6bdb5ca840aa46529e579c0306df2763868b2e9: Status 404 returned error can't find the container with id db37b952e28997d5bf70eba2d6bdb5ca840aa46529e579c0306df2763868b2e9
Jul 29 19:31:01.369864 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:01.369779    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6e2627_48d0_4286_a849_2d9c667293cc.slice/crio-9dfbe2faeaefaba6ac89af85f7840babd4f48107a7e0719044273035676b26c3 WatchSource:0}: Error finding container 9dfbe2faeaefaba6ac89af85f7840babd4f48107a7e0719044273035676b26c3: Status 404 returned error can't find the container with id 9dfbe2faeaefaba6ac89af85f7840babd4f48107a7e0719044273035676b26c3
Jul 29 19:31:01.940889 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.940840    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g24sx" event={"ID":"1fb6effe-4bf8-46eb-b57c-414367796ec5","Type":"ContainerStarted","Data":"49b8dcf8411ecc95ad886be91e51e9faada739e68b4075208e76a973b80f4e31"}
Jul 29 19:31:01.942545 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.942504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gr9w2" event={"ID":"e0d368bf-30b8-4d18-a17f-306eee453ddd","Type":"ContainerStarted","Data":"1df9ee7ba34479822cfcb403c1633c9e91952246332c930cccb2573fdec151f4"}
Jul 29 19:31:01.943911 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.943875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mck7p" event={"ID":"8b11f949-b457-40d7-832a-6c054be24568","Type":"ContainerStarted","Data":"db37b952e28997d5bf70eba2d6bdb5ca840aa46529e579c0306df2763868b2e9"}
Jul 29 19:31:01.945267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.945230    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5" event={"ID":"3b6e2627-48d0-4286-a849-2d9c667293cc","Type":"ContainerStarted","Data":"9dfbe2faeaefaba6ac89af85f7840babd4f48107a7e0719044273035676b26c3"}
Jul 29 19:31:01.946959 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.946932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666" event={"ID":"b4b31063-bda6-49f2-8c39-ab5a5b516391","Type":"ContainerStarted","Data":"2216a6c383e73cca5bbd7a919b945485cfcbfb2ce7bed5494bce81860a943291"}
Jul 29 19:31:01.947232 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.947185    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:31:01.948972 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.948928    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666"
Jul 29 19:31:01.968027 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:01.967980    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd7b6f779-b4666" podStartSLOduration=1.716835397 podStartE2EDuration="6.967965138s" totalImagesPullingTime="5.251129741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:01.966935147 +0000 UTC m=+67.925430068" watchObservedRunningTime="2026-07-29 19:31:01.967965138 +0000 UTC m=+67.926460032"
Jul 29 19:31:02.087694 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:02.087648    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:31:02.091038 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:02.090975    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/639a026f-f76d-4c50-ad25-20f56fe83839-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlkwn\" (UID: \"639a026f-f76d-4c50-ad25-20f56fe83839\") " pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:31:02.297425 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:02.297158    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8twzb\""
Jul 29 19:31:02.305881 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:02.305841    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlkwn"
Jul 29 19:31:02.716104 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:02.716067    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlkwn"]
Jul 29 19:31:02.858215 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:02.858132    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639a026f_f76d_4c50_ad25_20f56fe83839.slice/crio-5072bee2f84d7976fe22df222517851ecec00398530dcf2b873c08f35e3d2ec0 WatchSource:0}: Error finding container 5072bee2f84d7976fe22df222517851ecec00398530dcf2b873c08f35e3d2ec0: Status 404 returned error can't find the container with id 5072bee2f84d7976fe22df222517851ecec00398530dcf2b873c08f35e3d2ec0
Jul 29 19:31:02.952839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:02.952803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlkwn" event={"ID":"639a026f-f76d-4c50-ad25-20f56fe83839","Type":"ContainerStarted","Data":"5072bee2f84d7976fe22df222517851ecec00398530dcf2b873c08f35e3d2ec0"}
Jul 29 19:31:03.961557 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:03.961482    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g24sx" event={"ID":"1fb6effe-4bf8-46eb-b57c-414367796ec5","Type":"ContainerStarted","Data":"472a9b1bf21ac6187180b4b2366f09c3b759d204026e73c69e429941694f819d"}
Jul 29 19:31:04.968309 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:04.968243    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5" event={"ID":"3b6e2627-48d0-4286-a849-2d9c667293cc","Type":"ContainerStarted","Data":"d339033503ee078aac26cf8991e39551979bec6ca4c32c8e304e0a5c9eb9766d"}
Jul 29 19:31:04.968796 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:04.968322    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5" event={"ID":"3b6e2627-48d0-4286-a849-2d9c667293cc","Type":"ContainerStarted","Data":"ba6e6a67e15cc0d2d5b894ec6be9693d2bdfa1ffbde7d33bc8bec75a8b356b95"}
Jul 29 19:31:04.973892 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:04.973335    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g24sx" event={"ID":"1fb6effe-4bf8-46eb-b57c-414367796ec5","Type":"ContainerStarted","Data":"abceb9a3c73efef20714f53c9d84199a17de7370af099e542e70b4575ad8fcaa"}
Jul 29 19:31:05.027370 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:05.027323    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g24sx" podStartSLOduration=68.612886594 podStartE2EDuration="1m11.027306024s" totalImagesPullingTime="2.41441943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:05.026966831 +0000 UTC m=+70.985461726" watchObservedRunningTime="2026-07-29 19:31:05.027306024 +0000 UTC m=+70.985800918"
Jul 29 19:31:05.028328 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:05.028280    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-4zlg5" podStartSLOduration=3.50097829 podStartE2EDuration="6.028267828s" totalImagesPullingTime="2.527289538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:04.994549055 +0000 UTC m=+70.953043949" watchObservedRunningTime="2026-07-29 19:31:05.028267828 +0000 UTC m=+70.986762726"
Jul 29 19:31:07.407753 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.407711    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6lgvh"]
Jul 29 19:31:07.411509 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.410177    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.413225 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.412359    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qvrbw\""
Jul 29 19:31:07.413225 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.412667    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 19:31:07.413225 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.412830    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 19:31:07.413225 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.413040    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 19:31:07.505750 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.504383    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"]
Jul 29 19:31:07.507871 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.507727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.512368 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.511736    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 29 19:31:07.512368 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.512011    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d2wwp\""
Jul 29 19:31:07.512368 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.512203    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.539881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-root\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.539933    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.539967    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-sys\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.539990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-tls\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.540041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.540070    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blqx\" (UniqueName: \"kubernetes.io/projected/8e4b965e-52de-4fde-8617-68548bf1b8a1-kube-api-access-4blqx\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.540096    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-textfile\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.540126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-wtmp\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.540168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e4b965e-52de-4fde-8617-68548bf1b8a1-metrics-client-ca\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.540840 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.540796    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"]
Jul 29 19:31:07.641097 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c35df15c-f6ae-4903-850d-352c4f0d8171-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.641303 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641125    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641303 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641161    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blqx\" (UniqueName: \"kubernetes.io/projected/8e4b965e-52de-4fde-8617-68548bf1b8a1-kube-api-access-4blqx\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641303 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-textfile\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641303 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-wtmp\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641313    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.641436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e4b965e-52de-4fde-8617-68548bf1b8a1-metrics-client-ca\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-root\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641436 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641553 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6z8\" (UniqueName: \"kubernetes.io/projected/c35df15c-f6ae-4903-850d-352c4f0d8171-kube-api-access-vp6z8\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.641553 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-sys\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641553 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-tls\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641553 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.641553 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641526    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-textfile\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.641713 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.641691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-wtmp\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.642217 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.642199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e4b965e-52de-4fde-8617-68548bf1b8a1-metrics-client-ca\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.642274 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.642261    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-root\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.642485 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.642465    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e4b965e-52de-4fde-8617-68548bf1b8a1-sys\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.642613 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:31:07.642598    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 19:31:07.642680 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:31:07.642659    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-tls podName:8e4b965e-52de-4fde-8617-68548bf1b8a1 nodeName:}" failed. No retries permitted until 2026-07-29 19:31:08.142639466 +0000 UTC m=+74.101134339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-tls") pod "node-exporter-6lgvh" (UID: "8e4b965e-52de-4fde-8617-68548bf1b8a1") : secret "node-exporter-tls" not found
Jul 29 19:31:07.642680 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.642666    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.652968 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.652855    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.658951 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.658892    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blqx\" (UniqueName: \"kubernetes.io/projected/8e4b965e-52de-4fde-8617-68548bf1b8a1-kube-api-access-4blqx\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:07.742671 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.742636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6z8\" (UniqueName: \"kubernetes.io/projected/c35df15c-f6ae-4903-850d-352c4f0d8171-kube-api-access-vp6z8\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.742831 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.742706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.742831 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.742746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c35df15c-f6ae-4903-850d-352c4f0d8171-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.742831 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.742818    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.742984 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:31:07.742963    2555 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 29 19:31:07.743038 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:31:07.743032    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-tls podName:c35df15c-f6ae-4903-850d-352c4f0d8171 nodeName:}" failed. No retries permitted until 2026-07-29 19:31:08.243013258 +0000 UTC m=+74.201508133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-hv6qd" (UID: "c35df15c-f6ae-4903-850d-352c4f0d8171") : secret "openshift-state-metrics-tls" not found
Jul 29 19:31:07.744502 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.744471    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c35df15c-f6ae-4903-850d-352c4f0d8171-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.757800 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.757162    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:07.761161 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:07.761133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6z8\" (UniqueName: \"kubernetes.io/projected/c35df15c-f6ae-4903-850d-352c4f0d8171-kube-api-access-vp6z8\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:08.150824 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.150770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-tls\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:08.153657 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.153630    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e4b965e-52de-4fde-8617-68548bf1b8a1-node-exporter-tls\") pod \"node-exporter-6lgvh\" (UID: \"8e4b965e-52de-4fde-8617-68548bf1b8a1\") " pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:08.251983 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.251944    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:08.255385 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.255229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c35df15c-f6ae-4903-850d-352c4f0d8171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-hv6qd\" (UID: \"c35df15c-f6ae-4903-850d-352c4f0d8171\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:08.323347 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.322971    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lgvh"
Jul 29 19:31:08.421522 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.421430    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-775754dc7c-pqnnt"]
Jul 29 19:31:08.423284 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.423152    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"
Jul 29 19:31:08.426685 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.426657    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.429416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.428992    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 19:31:08.429416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.429021    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fwg7n\""
Jul 29 19:31:08.429416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.429139    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 19:31:08.429416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.429033    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 19:31:08.429416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.429034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 19:31:08.430138 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.430116    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 19:31:08.434333 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.434309    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 19:31:08.456905 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.456871    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-775754dc7c-pqnnt"]
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554200    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-kube-api-access-z8z4v\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-config\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-serving-cert\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-trusted-ca-bundle\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-oauth-serving-cert\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554425    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-oauth-config\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.554601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.554459    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-service-ca\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655488 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-oauth-serving-cert\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655509    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-oauth-config\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655547    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-service-ca\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-kube-api-access-z8z4v\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655645    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-config\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655703    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-serving-cert\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.655935 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.655744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-trusted-ca-bundle\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.656296 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.656264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-oauth-serving-cert\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.656662 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.656640    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-trusted-ca-bundle\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.656890 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.656866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-config\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.657469 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.657446    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-service-ca\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.660685 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.660658    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-serving-cert\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.665370 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.665348    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-oauth-config\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.672293 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.672229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-kube-api-access-z8z4v\") pod \"console-775754dc7c-pqnnt\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") " pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:08.740897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:08.740864    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:10.599781 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.599741    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69c757d79-g5bhl"]
Jul 29 19:31:10.603105 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.603076    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.605060    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.605320    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k97fp\""
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.605757    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.605921    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.606183    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f3sblsnl84k7k\""
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.606385    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 19:31:10.606613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.606530    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 19:31:10.622252 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.622228    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c757d79-g5bhl"]
Jul 29 19:31:10.674341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674258    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674341 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674344    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-grpc-tls\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-tls\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b892161e-6e80-46d2-bb45-64e5aefb98dc-metrics-client-ca\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.674565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.674541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdws\" (UniqueName: \"kubernetes.io/projected/b892161e-6e80-46d2-bb45-64e5aefb98dc-kube-api-access-rwdws\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.776718 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b892161e-6e80-46d2-bb45-64e5aefb98dc-metrics-client-ca\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.776903 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.776903 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdws\" (UniqueName: \"kubernetes.io/projected/b892161e-6e80-46d2-bb45-64e5aefb98dc-kube-api-access-rwdws\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.776903 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776837    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.776903 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.777137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776923    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-grpc-tls\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.777137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.776970    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.777137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.777010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-tls\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.780371 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.780339    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-tls\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.781096 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.781058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b892161e-6e80-46d2-bb45-64e5aefb98dc-metrics-client-ca\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.784830 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.784804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.784830 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.784817    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-grpc-tls\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.785011 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.784904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.785154 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.785119    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.786520 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.786496    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b892161e-6e80-46d2-bb45-64e5aefb98dc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.791701 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.791670    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdws\" (UniqueName: \"kubernetes.io/projected/b892161e-6e80-46d2-bb45-64e5aefb98dc-kube-api-access-rwdws\") pod \"thanos-querier-69c757d79-g5bhl\" (UID: \"b892161e-6e80-46d2-bb45-64e5aefb98dc\") " pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:10.914848 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:10.914803    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:11.000653 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:11.000614    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mck7p" event={"ID":"8b11f949-b457-40d7-832a-6c054be24568","Type":"ContainerStarted","Data":"9c32b1b07caef649308b76a18aeea68e9b9eacbe7b121386634cc514fb3ca12c"}
Jul 29 19:31:11.000840 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:11.000727    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:31:11.018863 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:11.018810    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mck7p" podStartSLOduration=72.128334018 podStartE2EDuration="1m17.018794899s" totalImagesPullingTime="4.890460881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:11.01770271 +0000 UTC m=+76.976197629" watchObservedRunningTime="2026-07-29 19:31:11.018794899 +0000 UTC m=+76.977289793"
Jul 29 19:31:12.045666 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.045632    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7665554cb6-zf7ln"]
Jul 29 19:31:12.048202 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.048181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.050784 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.050760    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 29 19:31:12.050913 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.050894    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mjlf8\""
Jul 29 19:31:12.050961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.050926    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 29 19:31:12.050997 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.050971    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 29 19:31:12.051193 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.051175    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7252a7kbvb8h8\""
Jul 29 19:31:12.051323 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.051183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 19:31:12.060739 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.060716    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7665554cb6-zf7ln"]
Jul 29 19:31:12.188775 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.188733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-secret-metrics-server-client-certs\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.188965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.188782    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eeb26ced-3607-457c-88f6-b7373a5206ef-metrics-server-audit-profiles\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.188965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.188823    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-secret-metrics-server-tls\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.188965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.188860    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eeb26ced-3607-457c-88f6-b7373a5206ef-audit-log\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.188965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.188896    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lc2f\" (UniqueName: \"kubernetes.io/projected/eeb26ced-3607-457c-88f6-b7373a5206ef-kube-api-access-8lc2f\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.189171 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.188975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb26ced-3607-457c-88f6-b7373a5206ef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.189171 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.189088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-client-ca-bundle\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290061 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290021    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-client-ca-bundle\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290097    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-secret-metrics-server-client-certs\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eeb26ced-3607-457c-88f6-b7373a5206ef-metrics-server-audit-profiles\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290148    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-secret-metrics-server-tls\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290179    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eeb26ced-3607-457c-88f6-b7373a5206ef-audit-log\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lc2f\" (UniqueName: \"kubernetes.io/projected/eeb26ced-3607-457c-88f6-b7373a5206ef-kube-api-access-8lc2f\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.290545 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290266    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb26ced-3607-457c-88f6-b7373a5206ef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.291073 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290986    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eeb26ced-3607-457c-88f6-b7373a5206ef-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.291073 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.290982    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eeb26ced-3607-457c-88f6-b7373a5206ef-audit-log\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.291259 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.291234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eeb26ced-3607-457c-88f6-b7373a5206ef-metrics-server-audit-profiles\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.293274 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.293227    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-client-ca-bundle\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.293932 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.293908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-secret-metrics-server-tls\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.294030 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.293932    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eeb26ced-3607-457c-88f6-b7373a5206ef-secret-metrics-server-client-certs\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.302715 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.302665    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lc2f\" (UniqueName: \"kubernetes.io/projected/eeb26ced-3607-457c-88f6-b7373a5206ef-kube-api-access-8lc2f\") pod \"metrics-server-7665554cb6-zf7ln\" (UID: \"eeb26ced-3607-457c-88f6-b7373a5206ef\") " pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.360965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.360733    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:12.608648 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:12.608553    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775754dc7c-pqnnt"]
Jul 29 19:31:13.779023 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.778985    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:31:13.783969 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.783938    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.786938 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.786523    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4bsjh\""
Jul 29 19:31:13.786938 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.786563    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 19:31:13.786938 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.786782    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 19:31:13.786938 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.786795    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 19:31:13.787437 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.787409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 19:31:13.787437 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.787421    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mj8r351hdbcj\""
Jul 29 19:31:13.787693 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.787676    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 19:31:13.787899 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.787807    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 19:31:13.787899 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.787816    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 19:31:13.787899 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.787870    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 19:31:13.788364 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.788257    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 19:31:13.790344 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.790114    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 19:31:13.790344 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.790132    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 19:31:13.806097 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.794683    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 19:31:13.806097 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.795432    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 19:31:13.809220 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.809190    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 19:31:13.812111 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.812087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:31:13.907118 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907301 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907301 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907301 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907301 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907241    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config-out\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907427    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907478    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907534    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907641    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-web-config\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907738    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.907778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.908077 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.908077 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:13.908077 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:13.907869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zdx\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-kube-api-access-87zdx\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.008854    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.008893    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.008919    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.008951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-web-config\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.008977    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009110    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009134    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zdx\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-kube-api-access-87zdx\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009210    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009327    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.011280 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config-out\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.012283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009384    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.012283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009423    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.012283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.009445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.012283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.010060    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.012283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.010708    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.012549 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.012421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.013957 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.013929    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.015311 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.015045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.015639 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.014563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.016381 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.016353    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.017005 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.016979    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.017966 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.017938    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.018208 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.018182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.021635 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.021610    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.025127 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.025104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.025331 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.025314    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config-out\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.025853 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.025830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.026244 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.026201    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.026906 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.026830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.026906 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.026858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.027060 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.026906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-web-config\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.027060 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.026953    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.029347 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.029300    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zdx\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-kube-api-access-87zdx\") pod \"prometheus-k8s-0\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:14.102205 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:14.102169    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:18.931508 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:18.931471    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-577699577-shb6z"
Jul 29 19:31:19.102897 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:19.102866    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4b965e_52de_4fde_8617_68548bf1b8a1.slice/crio-a607d2043ad5c2e3baaa1e88b8c370940459f293248df4cbf00d1ed0f611baad WatchSource:0}: Error finding container a607d2043ad5c2e3baaa1e88b8c370940459f293248df4cbf00d1ed0f611baad: Status 404 returned error can't find the container with id a607d2043ad5c2e3baaa1e88b8c370940459f293248df4cbf00d1ed0f611baad
Jul 29 19:31:19.313250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:19.313224    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c757d79-g5bhl"]
Jul 29 19:31:19.317751 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:19.317686    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb892161e_6e80_46d2_bb45_64e5aefb98dc.slice/crio-db4bd027fae20123a0d0d4270b01bad3cddf7d645d9c16763ddc8f89ab6e455f WatchSource:0}: Error finding container db4bd027fae20123a0d0d4270b01bad3cddf7d645d9c16763ddc8f89ab6e455f: Status 404 returned error can't find the container with id db4bd027fae20123a0d0d4270b01bad3cddf7d645d9c16763ddc8f89ab6e455f
Jul 29 19:31:19.389827 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:19.389753    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7665554cb6-zf7ln"]
Jul 29 19:31:19.394225 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:19.394202    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb26ced_3607_457c_88f6_b7373a5206ef.slice/crio-f807b96eaea95d2dfdd19e41ce7dc6cdeb37067a428d15520b5ced4fe1e5a7b2 WatchSource:0}: Error finding container f807b96eaea95d2dfdd19e41ce7dc6cdeb37067a428d15520b5ced4fe1e5a7b2: Status 404 returned error can't find the container with id f807b96eaea95d2dfdd19e41ce7dc6cdeb37067a428d15520b5ced4fe1e5a7b2
Jul 29 19:31:19.541963 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:19.541933    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd"]
Jul 29 19:31:19.543268 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:19.543247    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775754dc7c-pqnnt"]
Jul 29 19:31:19.544918 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:19.544889    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35df15c_f6ae_4903_850d_352c4f0d8171.slice/crio-76d5edefbf986f19289a5cd7e91eac81c566044782968d8b003f6f84228bc401 WatchSource:0}: Error finding container 76d5edefbf986f19289a5cd7e91eac81c566044782968d8b003f6f84228bc401: Status 404 returned error can't find the container with id 76d5edefbf986f19289a5cd7e91eac81c566044782968d8b003f6f84228bc401
Jul 29 19:31:19.546664 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:19.546634    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3681229_e8cc_41b0_a29c_3bdf72a86cb8.slice/crio-0cd4456c73eb1fcc28de02f314671fe0a2f31ab88c6266a2d48e0cd8e37143bc WatchSource:0}: Error finding container 0cd4456c73eb1fcc28de02f314671fe0a2f31ab88c6266a2d48e0cd8e37143bc: Status 404 returned error can't find the container with id 0cd4456c73eb1fcc28de02f314671fe0a2f31ab88c6266a2d48e0cd8e37143bc
Jul 29 19:31:19.548279 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:19.548240    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:31:19.551691 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:31:19.551609    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f99ac9_5d3b_4272_9b0b_1e83f9a92c38.slice/crio-a9856cfd8a4e01d0432505971c2ecd1fe72d95e2d1399932148df57509dc4cb5 WatchSource:0}: Error finding container a9856cfd8a4e01d0432505971c2ecd1fe72d95e2d1399932148df57509dc4cb5: Status 404 returned error can't find the container with id a9856cfd8a4e01d0432505971c2ecd1fe72d95e2d1399932148df57509dc4cb5
Jul 29 19:31:20.049569 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.049527    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gr9w2" event={"ID":"e0d368bf-30b8-4d18-a17f-306eee453ddd","Type":"ContainerStarted","Data":"509bb8e2a96b37ca93f6084204b3a7d57ba7b3287816a7327bd53f838c33840a"}
Jul 29 19:31:20.052359 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.052294    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775754dc7c-pqnnt" event={"ID":"d3681229-e8cc-41b0-a29c-3bdf72a86cb8","Type":"ContainerStarted","Data":"0cd4456c73eb1fcc28de02f314671fe0a2f31ab88c6266a2d48e0cd8e37143bc"}
Jul 29 19:31:20.055327 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.055293    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lgvh" event={"ID":"8e4b965e-52de-4fde-8617-68548bf1b8a1","Type":"ContainerStarted","Data":"a607d2043ad5c2e3baaa1e88b8c370940459f293248df4cbf00d1ed0f611baad"}
Jul 29 19:31:20.058283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.058153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd" event={"ID":"c35df15c-f6ae-4903-850d-352c4f0d8171","Type":"ContainerStarted","Data":"e4a9768002bb1fccc6c5cc7f042cea62c5f4a4fbbfc94adf29e8c1eb15a930f1"}
Jul 29 19:31:20.058283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.058208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd" event={"ID":"c35df15c-f6ae-4903-850d-352c4f0d8171","Type":"ContainerStarted","Data":"c9b4da682f91d2ce0f417bf49c6a2e5173955c673b0948ed6587d920bee5bb93"}
Jul 29 19:31:20.058283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.058235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd" event={"ID":"c35df15c-f6ae-4903-850d-352c4f0d8171","Type":"ContainerStarted","Data":"76d5edefbf986f19289a5cd7e91eac81c566044782968d8b003f6f84228bc401"}
Jul 29 19:31:20.061655 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.061554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"a9856cfd8a4e01d0432505971c2ecd1fe72d95e2d1399932148df57509dc4cb5"}
Jul 29 19:31:20.063705 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.063633    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlkwn" event={"ID":"639a026f-f76d-4c50-ad25-20f56fe83839","Type":"ContainerStarted","Data":"df9137c015fe68a41120ecdaec0b4c9c3ca24c6eb36d0a9e2a5895c3f4340909"}
Jul 29 19:31:20.069499 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.068804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xnzj9" event={"ID":"26f89cc3-3bde-4629-b636-f83c62f29c3f","Type":"ContainerStarted","Data":"a45ce0f7a226d77d39e2ab0495a31ab5c5615c5af85b8dc699b53d641cdad977"}
Jul 29 19:31:20.070467 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.070413    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:31:20.072758 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.071717    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gr9w2" podStartSLOduration=68.389131467 podStartE2EDuration="1m26.071700167s" totalImagesPullingTime="17.6825687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:20.07012188 +0000 UTC m=+86.028616798" watchObservedRunningTime="2026-07-29 19:31:20.071700167 +0000 UTC m=+86.030195063"
Jul 29 19:31:20.075393 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.075364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln" event={"ID":"eeb26ced-3607-457c-88f6-b7373a5206ef","Type":"ContainerStarted","Data":"f807b96eaea95d2dfdd19e41ce7dc6cdeb37067a428d15520b5ced4fe1e5a7b2"}
Jul 29 19:31:20.077003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.076976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"db4bd027fae20123a0d0d4270b01bad3cddf7d645d9c16763ddc8f89ab6e455f"}
Jul 29 19:31:20.084426 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.084372    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-xnzj9"
Jul 29 19:31:20.089682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:20.089425    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-xnzj9" podStartSLOduration=1.8406232660000001 podStartE2EDuration="25.089412601s" totalImagesPullingTime="23.248789335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:20.088791651 +0000 UTC m=+86.047286576" watchObservedRunningTime="2026-07-29 19:31:20.089412601 +0000 UTC m=+86.047907490"
Jul 29 19:31:30.136724 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.136633    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd" event={"ID":"c35df15c-f6ae-4903-850d-352c4f0d8171","Type":"ContainerStarted","Data":"4d6e094529634cf3cb225f02c0c44860f0a0902e5707e7c52866ae150f75f189"}
Jul 29 19:31:30.138723 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.138698    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="ac9c1462868657400ee351f79631a47cb9aa29c1373d34b366c147ae87945077" exitCode=0
Jul 29 19:31:30.138849 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.138775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"ac9c1462868657400ee351f79631a47cb9aa29c1373d34b366c147ae87945077"}
Jul 29 19:31:30.141394 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.141371    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlkwn" event={"ID":"639a026f-f76d-4c50-ad25-20f56fe83839","Type":"ContainerStarted","Data":"21673005c309fd577784c81a18b5718daac0d1aa5d2bbca3f5ee5ceb7d375531"}
Jul 29 19:31:30.143075 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.143050    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln" event={"ID":"eeb26ced-3607-457c-88f6-b7373a5206ef","Type":"ContainerStarted","Data":"94e5eab17c31ad4c42ac9737882a5c0fd4cda62995c1bd4638ad0959861b52ec"}
Jul 29 19:31:30.146506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.146478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"f6274c91bfe39a9520323565c4c767967f2c0423d6c4d412b8417cc3b5797128"}
Jul 29 19:31:30.146642 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.146616    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"29606e92f131682ad1935cb19bd5d87044938fe00d0539f3725468dec4b0b293"}
Jul 29 19:31:30.146708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.146684    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"ce9b897187a0a66e296a7f391f876f8eb3720e6f29a8887d1c064276eb25b550"}
Jul 29 19:31:30.148920 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.148894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775754dc7c-pqnnt" event={"ID":"d3681229-e8cc-41b0-a29c-3bdf72a86cb8","Type":"ContainerStarted","Data":"cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468"}
Jul 29 19:31:30.155082 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.155047    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lgvh" event={"ID":"8e4b965e-52de-4fde-8617-68548bf1b8a1","Type":"ContainerDied","Data":"5d04c62334efa20002aa89c4af97ea06e95658f95b32b86b8ce10f451b66960a"}
Jul 29 19:31:30.155186 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.155046    2555 generic.go:356] "Generic (PLEG): container finished" podID="8e4b965e-52de-4fde-8617-68548bf1b8a1" containerID="5d04c62334efa20002aa89c4af97ea06e95658f95b32b86b8ce10f451b66960a" exitCode=0
Jul 29 19:31:30.163288 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.163238    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-hv6qd" podStartSLOduration=13.418323264 podStartE2EDuration="23.163226304s" totalImagesPullingTime="9.74490304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:30.162337192 +0000 UTC m=+96.120832108" watchObservedRunningTime="2026-07-29 19:31:30.163226304 +0000 UTC m=+96.121721198"
Jul 29 19:31:30.246108 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.246056    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-775754dc7c-pqnnt" podStartSLOduration=12.344027535 podStartE2EDuration="22.246039223s" totalImagesPullingTime="9.902011688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:30.244871713 +0000 UTC m=+96.203366608" watchObservedRunningTime="2026-07-29 19:31:30.246039223 +0000 UTC m=+96.204534117"
Jul 29 19:31:30.324769 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:30.324723    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln" podStartSLOduration=8.269208078 podStartE2EDuration="18.324707507s" totalImagesPullingTime="10.055499429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:30.324247716 +0000 UTC m=+96.282742621" watchObservedRunningTime="2026-07-29 19:31:30.324707507 +0000 UTC m=+96.283202400"
Jul 29 19:31:31.164940 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:31.164794    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lgvh" event={"ID":"8e4b965e-52de-4fde-8617-68548bf1b8a1","Type":"ContainerStarted","Data":"84cab5847b79050d62d1434f323341f9f605f821ff0872c74b8c590ca2381dd6"}
Jul 29 19:31:31.164940 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:31.164867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lgvh" event={"ID":"8e4b965e-52de-4fde-8617-68548bf1b8a1","Type":"ContainerStarted","Data":"6e43f820512128a41ef15fcacc221e1238fc58cd8638d9047b1bdcf3d7fd8770"}
Jul 29 19:31:31.193420 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:31.188779    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6lgvh" podStartSLOduration=14.860337434 podStartE2EDuration="24.188753982s" totalImagesPullingTime="9.187204975s" totalInitContainerRuntime="141.211573ms" isStatefulPod=true podCreationTimestamp="2026-07-29 19:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:31.187682565 +0000 UTC m=+97.146177474" watchObservedRunningTime="2026-07-29 19:31:31.188753982 +0000 UTC m=+97.147248882"
Jul 29 19:31:32.174744 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.174648    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"ef9b7345b82ef7baf417b11581f135b75ab06782f0fcb42079f0f86fda94bfdb"}
Jul 29 19:31:32.175185 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.174773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"f03c73ecdc64e13777880805c4115d7e3f09df703309aa753d34494e93bba406"}
Jul 29 19:31:32.175185 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.174828    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:32.175185 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.174868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" event={"ID":"b892161e-6e80-46d2-bb45-64e5aefb98dc","Type":"ContainerStarted","Data":"2d264112a04576fc7fe5e97517346cc965d342d7cadcffd4309a53d4324e5fbe"}
Jul 29 19:31:32.203756 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.203698    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl" podStartSLOduration=10.46289338 podStartE2EDuration="22.203679739s" totalImagesPullingTime="11.740786359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:32.20278704 +0000 UTC m=+98.161281954" watchObservedRunningTime="2026-07-29 19:31:32.203679739 +0000 UTC m=+98.162174632"
Jul 29 19:31:32.361669 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.361628    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:32.361840 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:32.361701    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:36.193120 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:36.193076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"54c1aa471bda1865c6f64ebdf615f91141f92c96510a577689c81a6e3cc35c06"}
Jul 29 19:31:36.193468 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:36.193175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"0268df06482541ffc63108394f3ceab9edd9aeb36975dbb054d72e25a0764314"}
Jul 29 19:31:36.195329 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:36.195298    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlkwn" event={"ID":"639a026f-f76d-4c50-ad25-20f56fe83839","Type":"ContainerStarted","Data":"ae39ac6ca1fca50381e6cdbd538fd275be98fb4c5645c5c223eae7e2ca99039a"}
Jul 29 19:31:36.214686 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:36.214634    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dlkwn" podStartSLOduration=33.36326685 podStartE2EDuration="1m6.214616273s" totalImagesPullingTime="32.851349423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:30:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:36.21403401 +0000 UTC m=+102.172528934" watchObservedRunningTime="2026-07-29 19:31:36.214616273 +0000 UTC m=+102.173111162"
Jul 29 19:31:37.203270 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:37.203229    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"393acdf448ad87405bf8d10d91a3bb1176e95ca78373add8be45f5a4583615ab"}
Jul 29 19:31:37.203874 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:37.203307    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"3311ec9e02981f85669b93b4d60b3963a6f5813d9844256e1439a1fbdbee31f6"}
Jul 29 19:31:37.203874 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:37.203355    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"a71edc2898bc1057136913796acc9da53edc2da40252e452b1c1c28218d3858d"}
Jul 29 19:31:37.203874 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:37.203395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerStarted","Data":"1fa544abed5bf99889ff8c1e0ce0a651232a01009bba8b8389a714350b36a4de"}
Jul 29 19:31:37.232957 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:37.232899    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.589485045 podStartE2EDuration="24.232880001s" totalImagesPullingTime="15.61523873s" totalInitContainerRuntime="28.156226ms" isStatefulPod=false podCreationTimestamp="2026-07-29 19:31:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:31:37.230673737 +0000 UTC m=+103.189168646" watchObservedRunningTime="2026-07-29 19:31:37.232880001 +0000 UTC m=+103.191374895"
Jul 29 19:31:38.186323 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:38.186294    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69c757d79-g5bhl"
Jul 29 19:31:38.741769 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:38.741736    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:39.102394 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:39.102314    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:31:42.006809 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:42.006776    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mck7p"
Jul 29 19:31:52.366796 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:52.366769    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:52.370730 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:52.370706    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7665554cb6-zf7ln"
Jul 29 19:31:55.174081 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.174032    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-775754dc7c-pqnnt" podUID="d3681229-e8cc-41b0-a29c-3bdf72a86cb8" containerName="console" containerID="cri-o://cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468" gracePeriod=15
Jul 29 19:31:55.450792 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.450767    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:55.595786 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.595751    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-oauth-config\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.595970 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.595798    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-trusted-ca-bundle\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.595970 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.595824    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-kube-api-access-z8z4v\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.595970 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.595845    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-serving-cert\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.596136 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.595976    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-oauth-serving-cert\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.596136 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596032    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-service-ca\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.596248 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596150    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-config\") pod \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\" (UID: \"d3681229-e8cc-41b0-a29c-3bdf72a86cb8\") "
Jul 29 19:31:55.596340 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596312    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:31:55.596422 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596397    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:31:55.596646 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596617    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-config" (OuterVolumeSpecName: "console-config") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:31:55.596748 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596625    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-trusted-ca-bundle\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:55.596789 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.596617    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:31:55.598432 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.598400    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:31:55.598432 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.598419    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-kube-api-access-z8z4v" (OuterVolumeSpecName: "kube-api-access-z8z4v") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "kube-api-access-z8z4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:31:55.598553 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.598446    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3681229-e8cc-41b0-a29c-3bdf72a86cb8" (UID: "d3681229-e8cc-41b0-a29c-3bdf72a86cb8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:31:55.697391 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.697365    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-config\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:55.697391 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.697388    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-oauth-config\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:55.697549 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.697398    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-kube-api-access-z8z4v\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:55.697549 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.697407    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-console-serving-cert\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:55.697549 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.697414    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-oauth-serving-cert\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:55.697549 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:55.697422    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3681229-e8cc-41b0-a29c-3bdf72a86cb8-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:31:56.278598 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.278548    2555 generic.go:356] "Generic (PLEG): container finished" podID="d3681229-e8cc-41b0-a29c-3bdf72a86cb8" containerID="cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468" exitCode=0
Jul 29 19:31:56.279033 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.278606    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-775754dc7c-pqnnt"
Jul 29 19:31:56.279033 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.278615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775754dc7c-pqnnt" event={"ID":"d3681229-e8cc-41b0-a29c-3bdf72a86cb8","Type":"ContainerDied","Data":"cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468"}
Jul 29 19:31:56.279033 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.278670    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-775754dc7c-pqnnt" event={"ID":"d3681229-e8cc-41b0-a29c-3bdf72a86cb8","Type":"ContainerDied","Data":"0cd4456c73eb1fcc28de02f314671fe0a2f31ab88c6266a2d48e0cd8e37143bc"}
Jul 29 19:31:56.279033 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.278689    2555 scope.go:117] "RemoveContainer" containerID="cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468"
Jul 29 19:31:56.287851 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.287830    2555 scope.go:117] "RemoveContainer" containerID="cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468"
Jul 29 19:31:56.288138 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:31:56.288117    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468\": container with ID starting with cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468 not found: ID does not exist" containerID="cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468"
Jul 29 19:31:56.288189 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.288149    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468"} err="failed to get container status \"cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468\": rpc error: code = NotFound desc = could not find container \"cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468\": container with ID starting with cad7e15e0bcd9a05fe7849b9be2fd78a333a558e0e019414a30ef5fbabed0468 not found: ID does not exist"
Jul 29 19:31:56.306300 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.306269    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-775754dc7c-pqnnt"]
Jul 29 19:31:56.313718 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.313695    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-775754dc7c-pqnnt"]
Jul 29 19:31:56.657854 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:31:56.657777    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3681229-e8cc-41b0-a29c-3bdf72a86cb8" path="/var/lib/kubelet/pods/d3681229-e8cc-41b0-a29c-3bdf72a86cb8/volumes"
Jul 29 19:32:14.102630 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:14.102566    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:14.152428 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:14.152399    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:14.375271 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:14.375193    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:32.294644 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.294600    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:32:32.295241 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.295025    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="prometheus" containerID="cri-o://0268df06482541ffc63108394f3ceab9edd9aeb36975dbb054d72e25a0764314" gracePeriod=600
Jul 29 19:32:32.295241 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.295041    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy" containerID="cri-o://3311ec9e02981f85669b93b4d60b3963a6f5813d9844256e1439a1fbdbee31f6" gracePeriod=600
Jul 29 19:32:32.295241 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.295078    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-thanos" containerID="cri-o://393acdf448ad87405bf8d10d91a3bb1176e95ca78373add8be45f5a4583615ab" gracePeriod=600
Jul 29 19:32:32.295241 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.295126    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-web" containerID="cri-o://a71edc2898bc1057136913796acc9da53edc2da40252e452b1c1c28218d3858d" gracePeriod=600
Jul 29 19:32:32.295241 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.295085    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="config-reloader" containerID="cri-o://54c1aa471bda1865c6f64ebdf615f91141f92c96510a577689c81a6e3cc35c06" gracePeriod=600
Jul 29 19:32:32.295241 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.295105    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="thanos-sidecar" containerID="cri-o://1fa544abed5bf99889ff8c1e0ce0a651232a01009bba8b8389a714350b36a4de" gracePeriod=600
Jul 29 19:32:32.426162 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426134    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="393acdf448ad87405bf8d10d91a3bb1176e95ca78373add8be45f5a4583615ab" exitCode=0
Jul 29 19:32:32.426162 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426157    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="3311ec9e02981f85669b93b4d60b3963a6f5813d9844256e1439a1fbdbee31f6" exitCode=0
Jul 29 19:32:32.426162 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426164    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="1fa544abed5bf99889ff8c1e0ce0a651232a01009bba8b8389a714350b36a4de" exitCode=0
Jul 29 19:32:32.426162 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426169    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="54c1aa471bda1865c6f64ebdf615f91141f92c96510a577689c81a6e3cc35c06" exitCode=0
Jul 29 19:32:32.426384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426174    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="0268df06482541ffc63108394f3ceab9edd9aeb36975dbb054d72e25a0764314" exitCode=0
Jul 29 19:32:32.426384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"393acdf448ad87405bf8d10d91a3bb1176e95ca78373add8be45f5a4583615ab"}
Jul 29 19:32:32.426384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426277    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"3311ec9e02981f85669b93b4d60b3963a6f5813d9844256e1439a1fbdbee31f6"}
Jul 29 19:32:32.426384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"1fa544abed5bf99889ff8c1e0ce0a651232a01009bba8b8389a714350b36a4de"}
Jul 29 19:32:32.426384 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"54c1aa471bda1865c6f64ebdf615f91141f92c96510a577689c81a6e3cc35c06"}
Jul 29 19:32:32.426529 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:32.426389    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"0268df06482541ffc63108394f3ceab9edd9aeb36975dbb054d72e25a0764314"}
Jul 29 19:32:33.435429 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.435398    2555 generic.go:356] "Generic (PLEG): container finished" podID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerID="a71edc2898bc1057136913796acc9da53edc2da40252e452b1c1c28218d3858d" exitCode=0
Jul 29 19:32:33.435794 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.435446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"a71edc2898bc1057136913796acc9da53edc2da40252e452b1c1c28218d3858d"}
Jul 29 19:32:33.545225 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.545197    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:33.625883 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.625846    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-kube-rbac-proxy\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626062 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.625894    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626062 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.625924    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-serving-certs-ca-bundle\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626062 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.625953    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-1\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626062 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.625989    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-kubelet-serving-ca-bundle\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626062 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626036    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-metrics-client-ca\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626067    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-trusted-ca-bundle\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626094    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-metrics-client-certs\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-db\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626152    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-thanos-prometheus-http-client-file\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626176    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-tls\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626212    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-grpc-tls\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626242    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-2\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626271    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-web-config\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626321    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-0\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626309    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626354    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-tls-assets\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626384    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config-out\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626418    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626456    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626483    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87zdx\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-kube-api-access-87zdx\") pod \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\" (UID: \"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38\") "
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626797    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626884    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.626961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.626900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.627399 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.627170    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.627399 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.627289    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.628884 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.628852    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.629209 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.629098    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:32:33.629381 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.629327    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.629381 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.629351    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.629711 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.629685    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:32:33.632322 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.632277    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-kube-api-access-87zdx" (OuterVolumeSpecName: "kube-api-access-87zdx") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "kube-api-access-87zdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:33.632322 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.632300    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.632483 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.632412    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config-out" (OuterVolumeSpecName: "config-out") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:32:33.632483 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.632444    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:32:33.632630 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.632503    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.632687 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.632668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.633322 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.633299    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config" (OuterVolumeSpecName: "config") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.633457 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.633414    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.634353 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.634326    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.642247 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.642222    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-web-config" (OuterVolumeSpecName: "web-config") pod "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" (UID: "91f99ac9-5d3b-4272-9b0b-1e83f9a92c38"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:32:33.727949 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727917    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.727949 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727942    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727959    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727972    2555 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-configmap-metrics-client-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727981    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727991    2555 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-metrics-client-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.727999    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-db\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728009    2555 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728023    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728036    2555 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-grpc-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728050    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728063    2555 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-web-config\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728077    2555 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728089    2555 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-tls-assets\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728103    2555 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config-out\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728115    2555 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-config\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728128    2555 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728144    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87zdx\" (UniqueName: \"kubernetes.io/projected/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-kube-api-access-87zdx\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:33.728180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:33.728159    2555 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38-secret-kube-rbac-proxy\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:32:34.442246 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.442214    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"91f99ac9-5d3b-4272-9b0b-1e83f9a92c38","Type":"ContainerDied","Data":"a9856cfd8a4e01d0432505971c2ecd1fe72d95e2d1399932148df57509dc4cb5"}
Jul 29 19:32:34.442246 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.442231    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.442699 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.442290    2555 scope.go:117] "RemoveContainer" containerID="393acdf448ad87405bf8d10d91a3bb1176e95ca78373add8be45f5a4583615ab"
Jul 29 19:32:34.450447 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.450425    2555 scope.go:117] "RemoveContainer" containerID="3311ec9e02981f85669b93b4d60b3963a6f5813d9844256e1439a1fbdbee31f6"
Jul 29 19:32:34.458340 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.458320    2555 scope.go:117] "RemoveContainer" containerID="a71edc2898bc1057136913796acc9da53edc2da40252e452b1c1c28218d3858d"
Jul 29 19:32:34.464856 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.464839    2555 scope.go:117] "RemoveContainer" containerID="1fa544abed5bf99889ff8c1e0ce0a651232a01009bba8b8389a714350b36a4de"
Jul 29 19:32:34.468965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.468940    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:32:34.473388 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.473370    2555 scope.go:117] "RemoveContainer" containerID="54c1aa471bda1865c6f64ebdf615f91141f92c96510a577689c81a6e3cc35c06"
Jul 29 19:32:34.474199 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.474178    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:32:34.480219 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.480204    2555 scope.go:117] "RemoveContainer" containerID="0268df06482541ffc63108394f3ceab9edd9aeb36975dbb054d72e25a0764314"
Jul 29 19:32:34.487336 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.487320    2555 scope.go:117] "RemoveContainer" containerID="ac9c1462868657400ee351f79631a47cb9aa29c1373d34b366c147ae87945077"
Jul 29 19:32:34.504483 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504461    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:32:34.504760 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504747    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-thanos"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504763    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-thanos"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504774    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-web"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504780    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-web"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504787    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="config-reloader"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504793    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="config-reloader"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504801    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="prometheus"
Jul 29 19:32:34.504810 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504806    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="prometheus"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504813    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="thanos-sidecar"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504819    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="thanos-sidecar"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504825    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504830    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504840    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="init-config-reloader"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504845    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="init-config-reloader"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504857    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3681229-e8cc-41b0-a29c-3bdf72a86cb8" containerName="console"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504863    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3681229-e8cc-41b0-a29c-3bdf72a86cb8" containerName="console"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504902    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="prometheus"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504910    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-web"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504916    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3681229-e8cc-41b0-a29c-3bdf72a86cb8" containerName="console"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504921    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy-thanos"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504927    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="kube-rbac-proxy"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504932    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="config-reloader"
Jul 29 19:32:34.505003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.504939    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" containerName="thanos-sidecar"
Jul 29 19:32:34.509232 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.509212    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.511427 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511394    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 19:32:34.511761 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511743    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4bsjh\""
Jul 29 19:32:34.511848 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511747    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 19:32:34.511848 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511812    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mj8r351hdbcj\""
Jul 29 19:32:34.511848 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511831    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 19:32:34.511999 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 19:32:34.511999 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.511886    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 19:32:34.512094 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512039    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 19:32:34.512147 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512110    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 19:32:34.512147 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512129    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 19:32:34.512227 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512156    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 19:32:34.512270 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512236    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 19:32:34.512483 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512463    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 19:32:34.512546 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.512480    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 19:32:34.514194 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.514177    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 19:32:34.520434 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.520407    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 19:32:34.524159 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.524137    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:32:34.632647 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632647 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632650    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632676    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632735    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcd3fba0-1090-4d47-b41a-97380b98ff69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632818    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.632844 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632833    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdk5\" (UniqueName: \"kubernetes.io/projected/bcd3fba0-1090-4d47-b41a-97380b98ff69-kube-api-access-ltdk5\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632850    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-web-config\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632877    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-config\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632956    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.632999    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633038    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcd3fba0-1090-4d47-b41a-97380b98ff69-config-out\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633113    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633312 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.633312 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.633197    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.657405 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.657340    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f99ac9-5d3b-4272-9b0b-1e83f9a92c38" path="/var/lib/kubelet/pods/91f99ac9-5d3b-4272-9b0b-1e83f9a92c38/volumes"
Jul 29 19:32:34.733933 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.733903    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.733933 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.733935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdk5\" (UniqueName: \"kubernetes.io/projected/bcd3fba0-1090-4d47-b41a-97380b98ff69-kube-api-access-ltdk5\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.733952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-web-config\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.733969    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-config\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734004    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734094    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734367 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcd3fba0-1090-4d47-b41a-97380b98ff69-config-out\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.734901 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735193 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734929    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735193 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.734946    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735297 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.735192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735297 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.735192    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735979 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.735425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735979 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.735453    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.735979 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.735957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.736664 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.736199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.736664 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.736249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.736664 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.736284    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.737287 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.736313    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcd3fba0-1090-4d47-b41a-97380b98ff69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.737287 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.736919    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.737287 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.737103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.738295 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.738266    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.738718 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.738695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-web-config\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.738932 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.738879    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.739012 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.738949    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.739289 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.739252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.740230 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.740205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bcd3fba0-1090-4d47-b41a-97380b98ff69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.740331 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.740239    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bcd3fba0-1090-4d47-b41a-97380b98ff69-config-out\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.740331 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.740320    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bcd3fba0-1090-4d47-b41a-97380b98ff69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.740434 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.740336    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-config\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.740503 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.740486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.740660 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.740641    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.741403 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.741385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bcd3fba0-1090-4d47-b41a-97380b98ff69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.744970 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.744951    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdk5\" (UniqueName: \"kubernetes.io/projected/bcd3fba0-1090-4d47-b41a-97380b98ff69-kube-api-access-ltdk5\") pod \"prometheus-k8s-0\" (UID: \"bcd3fba0-1090-4d47-b41a-97380b98ff69\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.824038 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.824002    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:32:34.970532 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:34.970480    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 19:32:34.976483 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:32:34.976455    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd3fba0_1090_4d47_b41a_97380b98ff69.slice/crio-7738b89c0175608fe7f60f70fc4f5684f9b9ef7cb39227249f8c1346a273a6ac WatchSource:0}: Error finding container 7738b89c0175608fe7f60f70fc4f5684f9b9ef7cb39227249f8c1346a273a6ac: Status 404 returned error can't find the container with id 7738b89c0175608fe7f60f70fc4f5684f9b9ef7cb39227249f8c1346a273a6ac
Jul 29 19:32:35.448861 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:35.448821    2555 generic.go:356] "Generic (PLEG): container finished" podID="bcd3fba0-1090-4d47-b41a-97380b98ff69" containerID="05feacc77bf7aeaeccbc09393fdc02a842e1507111ce35a29ee840fb32536f6a" exitCode=0
Jul 29 19:32:35.449332 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:35.448907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerDied","Data":"05feacc77bf7aeaeccbc09393fdc02a842e1507111ce35a29ee840fb32536f6a"}
Jul 29 19:32:35.449332 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:35.448971    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"7738b89c0175608fe7f60f70fc4f5684f9b9ef7cb39227249f8c1346a273a6ac"}
Jul 29 19:32:36.456231 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.456194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"15d8bbb3df637d75820a536cc42316f0f2f82786bfcf221a05f72ebf6ef0c850"}
Jul 29 19:32:36.456633 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.456289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"3e0e2e56726e9ebef84cf361bdbd1fdbd967a498782da67cad11c176202a4867"}
Jul 29 19:32:36.456633 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.456318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"7b9d2be00dd5213ab4a5bc25b050fcc18233a17cef30fde6f568086978f14289"}
Jul 29 19:32:36.456633 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.456340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"f308c50b58fb1e29c34b18395c016df57bdcceee7bc44b0aa263f2e12ee2b0e5"}
Jul 29 19:32:36.456633 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.456374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"0db5937a49f4dc621d7b6605d8afe1d6a763072df582ebffed6965d583967511"}
Jul 29 19:32:36.456633 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.456401    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bcd3fba0-1090-4d47-b41a-97380b98ff69","Type":"ContainerStarted","Data":"8f45a8c75fce4e7536865addcbdc9e623a2b41379e2370377d8aec2ae92dc44a"}
Jul 29 19:32:36.487543 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:36.487500    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.466263135 podStartE2EDuration="2.487486392s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.223257ms" isStatefulPod=false podCreationTimestamp="2026-07-29 19:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:32:36.485902236 +0000 UTC m=+162.444397152" watchObservedRunningTime="2026-07-29 19:32:36.487486392 +0000 UTC m=+162.445981285"
Jul 29 19:32:39.824408 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:32:39.824374    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:33:34.825033 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:33:34.824993    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:33:34.841054 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:33:34.841021    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:33:35.698792 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:33:35.698765    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 19:34:54.472170 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:34:54.472141    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:34:54.476173 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:34:54.476146    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 19:34:54.479275 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:34:54.479253    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:35:39.813262 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.813226    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"]
Jul 29 19:35:39.816546 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.816531    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.819456 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.819402    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 19:35:39.819456 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.819425    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 19:35:39.819696 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.819402    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 19:35:39.819696 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.819496    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2vqxw\""
Jul 29 19:35:39.819861 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.819831    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 19:35:39.834259 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.834232    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"]
Jul 29 19:35:39.884320 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.884282    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/684e45cf-b6db-448c-b7ce-73fd08a66308-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.884494 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.884331    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkvl\" (UniqueName: \"kubernetes.io/projected/684e45cf-b6db-448c-b7ce-73fd08a66308-kube-api-access-4hkvl\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.884494 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.884352    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/684e45cf-b6db-448c-b7ce-73fd08a66308-tmp\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.884494 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.884439    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/684e45cf-b6db-448c-b7ce-73fd08a66308-manifests\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.884645 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.884514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/684e45cf-b6db-448c-b7ce-73fd08a66308-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.985851 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.985816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/684e45cf-b6db-448c-b7ce-73fd08a66308-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.986039 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.985862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkvl\" (UniqueName: \"kubernetes.io/projected/684e45cf-b6db-448c-b7ce-73fd08a66308-kube-api-access-4hkvl\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.986039 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.985895    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/684e45cf-b6db-448c-b7ce-73fd08a66308-tmp\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.986039 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.985932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/684e45cf-b6db-448c-b7ce-73fd08a66308-manifests\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.986039 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.985986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/684e45cf-b6db-448c-b7ce-73fd08a66308-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.986386 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.986358    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/684e45cf-b6db-448c-b7ce-73fd08a66308-manifests\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.988408 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.988375    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/684e45cf-b6db-448c-b7ce-73fd08a66308-tmp\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.988506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.988452    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/684e45cf-b6db-448c-b7ce-73fd08a66308-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.988506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.988483    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/684e45cf-b6db-448c-b7ce-73fd08a66308-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:39.996300 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:39.996276    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkvl\" (UniqueName: \"kubernetes.io/projected/684e45cf-b6db-448c-b7ce-73fd08a66308-kube-api-access-4hkvl\") pod \"opendatahub-operator-controller-manager-5d75b7b96-lbglf\" (UID: \"684e45cf-b6db-448c-b7ce-73fd08a66308\") " pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:40.127269 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:40.127190    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:40.266565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:40.266540    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"]
Jul 29 19:35:40.268877 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:35:40.268850    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684e45cf_b6db_448c_b7ce_73fd08a66308.slice/crio-c2b48f7e156f4d815ac80e106c3e167dfc162389390c38bdb913cbe683c10e64 WatchSource:0}: Error finding container c2b48f7e156f4d815ac80e106c3e167dfc162389390c38bdb913cbe683c10e64: Status 404 returned error can't find the container with id c2b48f7e156f4d815ac80e106c3e167dfc162389390c38bdb913cbe683c10e64
Jul 29 19:35:40.270723 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:40.270703    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:35:41.136964 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:41.136915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf" event={"ID":"684e45cf-b6db-448c-b7ce-73fd08a66308","Type":"ContainerStarted","Data":"c2b48f7e156f4d815ac80e106c3e167dfc162389390c38bdb913cbe683c10e64"}
Jul 29 19:35:44.153653 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:44.153613    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf" event={"ID":"684e45cf-b6db-448c-b7ce-73fd08a66308","Type":"ContainerStarted","Data":"a9ebabb874b3381ef8b7c83bedb3cad36a5d3d9830571f3c185540517fea414a"}
Jul 29 19:35:45.154138 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:35:45.154107    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684e45cf_b6db_448c_b7ce_73fd08a66308.slice/crio-a9ebabb874b3381ef8b7c83bedb3cad36a5d3d9830571f3c185540517fea414a.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 19:35:45.159198 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:45.159170    2555 generic.go:356] "Generic (PLEG): container finished" podID="684e45cf-b6db-448c-b7ce-73fd08a66308" containerID="a9ebabb874b3381ef8b7c83bedb3cad36a5d3d9830571f3c185540517fea414a" exitCode=0
Jul 29 19:35:45.159332 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:45.159233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf" event={"ID":"684e45cf-b6db-448c-b7ce-73fd08a66308","Type":"ContainerDied","Data":"a9ebabb874b3381ef8b7c83bedb3cad36a5d3d9830571f3c185540517fea414a"}
Jul 29 19:35:46.166134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:46.166094    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf" event={"ID":"684e45cf-b6db-448c-b7ce-73fd08a66308","Type":"ContainerStarted","Data":"964ce38668485b9024e5f58fa9a81493e827a7016315d30fe4e60b42d81a16e7"}
Jul 29 19:35:46.166630 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:46.166233    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:46.203561 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:46.195922    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf" podStartSLOduration=2.072247773 podStartE2EDuration="7.195896173s" totalImagesPullingTime="4.090855016s" totalInitContainerRuntime="1.032793384s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:35:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:35:46.195321581 +0000 UTC m=+352.153816479" watchObservedRunningTime="2026-07-29 19:35:46.195896173 +0000 UTC m=+352.154391068"
Jul 29 19:35:57.173016 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:57.172983    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d75b7b96-lbglf"
Jul 29 19:35:58.547145 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.547110    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"]
Jul 29 19:35:58.556712 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.556691    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.559330 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.559101    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 19:35:58.559487 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.559332    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bnvdj\""
Jul 29 19:35:58.559487 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.559373    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 19:35:58.559794 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.559775    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 19:35:58.559975 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.559955    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 19:35:58.560078 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.560058    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 19:35:58.560724 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.560707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"]
Jul 29 19:35:58.640942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.640900    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45280a7d-00b8-42d6-a6d9-66d239c2ea22-metrics-cert\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.640942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.640945    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7cnh\" (UniqueName: \"kubernetes.io/projected/45280a7d-00b8-42d6-a6d9-66d239c2ea22-kube-api-access-f7cnh\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.641147 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.640992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45280a7d-00b8-42d6-a6d9-66d239c2ea22-cert\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.641147 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.641049    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45280a7d-00b8-42d6-a6d9-66d239c2ea22-manager-config\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.741936 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.741901    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45280a7d-00b8-42d6-a6d9-66d239c2ea22-metrics-cert\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.742109 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.741946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cnh\" (UniqueName: \"kubernetes.io/projected/45280a7d-00b8-42d6-a6d9-66d239c2ea22-kube-api-access-f7cnh\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.742109 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.741972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45280a7d-00b8-42d6-a6d9-66d239c2ea22-cert\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.742109 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.742003    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45280a7d-00b8-42d6-a6d9-66d239c2ea22-manager-config\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.742798 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.742773    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/45280a7d-00b8-42d6-a6d9-66d239c2ea22-manager-config\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.744651 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.744631    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/45280a7d-00b8-42d6-a6d9-66d239c2ea22-cert\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.744762 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.744690    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/45280a7d-00b8-42d6-a6d9-66d239c2ea22-metrics-cert\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.750543 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.750515    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cnh\" (UniqueName: \"kubernetes.io/projected/45280a7d-00b8-42d6-a6d9-66d239c2ea22-kube-api-access-f7cnh\") pod \"lws-controller-manager-fcbbc6665-ngk5f\" (UID: \"45280a7d-00b8-42d6-a6d9-66d239c2ea22\") " pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:58.867037 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:58.866963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:35:59.017014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.016991    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"]
Jul 29 19:35:59.020280 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:35:59.020249    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45280a7d_00b8_42d6_a6d9_66d239c2ea22.slice/crio-079dfc5dd7cff6b80c353396c90603aa655ce256e97d5e04a76b0a60bca9cb68 WatchSource:0}: Error finding container 079dfc5dd7cff6b80c353396c90603aa655ce256e97d5e04a76b0a60bca9cb68: Status 404 returned error can't find the container with id 079dfc5dd7cff6b80c353396c90603aa655ce256e97d5e04a76b0a60bca9cb68
Jul 29 19:35:59.220292 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.220259    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f" event={"ID":"45280a7d-00b8-42d6-a6d9-66d239c2ea22","Type":"ContainerStarted","Data":"079dfc5dd7cff6b80c353396c90603aa655ce256e97d5e04a76b0a60bca9cb68"}
Jul 29 19:35:59.925008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.924970    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"]
Jul 29 19:35:59.928147 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.928127    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:35:59.930287 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.930270    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 19:35:59.930994 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.930972    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 19:35:59.931083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.930972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 19:35:59.931083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.931013    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-j8k7c\""
Jul 29 19:35:59.931083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.930998    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 19:35:59.940642 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:35:59.940621    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"]
Jul 29 19:36:00.053457 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.053420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e686ba-1b46-4989-b6e5-ba3adec565c4-tmp\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.053646 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.053470    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e686ba-1b46-4989-b6e5-ba3adec565c4-tls-certs\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.053646 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.053574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grtt\" (UniqueName: \"kubernetes.io/projected/c9e686ba-1b46-4989-b6e5-ba3adec565c4-kube-api-access-9grtt\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.154365 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.154329    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grtt\" (UniqueName: \"kubernetes.io/projected/c9e686ba-1b46-4989-b6e5-ba3adec565c4-kube-api-access-9grtt\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.154602 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.154446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e686ba-1b46-4989-b6e5-ba3adec565c4-tmp\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.154602 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.154478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e686ba-1b46-4989-b6e5-ba3adec565c4-tls-certs\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.157273 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.157238    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e686ba-1b46-4989-b6e5-ba3adec565c4-tmp\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.157433 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.157413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e686ba-1b46-4989-b6e5-ba3adec565c4-tls-certs\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.170217 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.170191    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grtt\" (UniqueName: \"kubernetes.io/projected/c9e686ba-1b46-4989-b6e5-ba3adec565c4-kube-api-access-9grtt\") pod \"kube-auth-proxy-6cc65fd5df-njhfl\" (UID: \"c9e686ba-1b46-4989-b6e5-ba3adec565c4\") " pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.238434 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.238406    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"
Jul 29 19:36:00.403599 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:00.403547    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl"]
Jul 29 19:36:00.406412 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:36:00.406381    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e686ba_1b46_4989_b6e5_ba3adec565c4.slice/crio-f7e095df31817c56152cc9d7604765de368d9058bb0758e5bd671cab6839b0df WatchSource:0}: Error finding container f7e095df31817c56152cc9d7604765de368d9058bb0758e5bd671cab6839b0df: Status 404 returned error can't find the container with id f7e095df31817c56152cc9d7604765de368d9058bb0758e5bd671cab6839b0df
Jul 29 19:36:01.237801 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:01.237708    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl" event={"ID":"c9e686ba-1b46-4989-b6e5-ba3adec565c4","Type":"ContainerStarted","Data":"f7e095df31817c56152cc9d7604765de368d9058bb0758e5bd671cab6839b0df"}
Jul 29 19:36:04.254671 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:04.254631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f" event={"ID":"45280a7d-00b8-42d6-a6d9-66d239c2ea22","Type":"ContainerStarted","Data":"40f0d6292e77458e87e38adbd96089ed9d6e8d9511f35bb583f75de6e3a02f82"}
Jul 29 19:36:04.255148 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:04.254710    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:36:04.256183 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:04.256149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl" event={"ID":"c9e686ba-1b46-4989-b6e5-ba3adec565c4","Type":"ContainerStarted","Data":"d1fa0b67613ef0c34f2824fee4edded532174a25301a1ec8c7fa4bef24491b54"}
Jul 29 19:36:04.275308 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:04.275264    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f" podStartSLOduration=1.897386119 podStartE2EDuration="6.275250614s" totalImagesPullingTime="4.377864495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:36:04.274364927 +0000 UTC m=+370.232859839" watchObservedRunningTime="2026-07-29 19:36:04.275250614 +0000 UTC m=+370.233745508"
Jul 29 19:36:04.291986 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:04.291941    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cc65fd5df-njhfl" podStartSLOduration=2.266494516 podStartE2EDuration="5.291926281s" totalImagesPullingTime="3.025431765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:36:04.291102225 +0000 UTC m=+370.249597119" watchObservedRunningTime="2026-07-29 19:36:04.291926281 +0000 UTC m=+370.250421176"
Jul 29 19:36:15.262737 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:36:15.262703    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fcbbc6665-ngk5f"
Jul 29 19:37:48.339636 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.339600    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"]
Jul 29 19:37:48.342189 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.342169    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.344210 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.344189    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 19:37:48.344321 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.344273    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h7xjc\""
Jul 29 19:37:48.344321 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.344278    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 19:37:48.344441 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.344279    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 19:37:48.344665 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.344641    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 19:37:48.351778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.351756    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"]
Jul 29 19:37:48.485574 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.485536    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.485574 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.485608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.485854 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.485722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4n8\" (UniqueName: \"kubernetes.io/projected/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-kube-api-access-ps4n8\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.586983 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.586941    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.587163 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.586987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.587163 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.587055    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4n8\" (UniqueName: \"kubernetes.io/projected/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-kube-api-access-ps4n8\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.587163 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:37:48.587153    2555 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 29 19:37:48.587318 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:37:48.587216    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-plugin-serving-cert podName:cd7e4b4c-cf34-4858-85f3-190ffe31edc2 nodeName:}" failed. No retries permitted until 2026-07-29 19:37:49.087200438 +0000 UTC m=+475.045695310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-x29j4" (UID: "cd7e4b4c-cf34-4858-85f3-190ffe31edc2") : secret "plugin-serving-cert" not found
Jul 29 19:37:48.587533 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.587513    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:48.602369 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:48.602308    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4n8\" (UniqueName: \"kubernetes.io/projected/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-kube-api-access-ps4n8\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:49.092408 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:49.092345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:49.094963 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:49.094943    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7e4b4c-cf34-4858-85f3-190ffe31edc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-x29j4\" (UID: \"cd7e4b4c-cf34-4858-85f3-190ffe31edc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:49.263637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:49.263566    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"
Jul 29 19:37:49.389267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:49.389075    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4"]
Jul 29 19:37:49.391943 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:37:49.391917    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd7e4b4c_cf34_4858_85f3_190ffe31edc2.slice/crio-a36cb67969cb777b2ba514204a8864fb852d71e1ddaf4a165ecbfa265230a1fe WatchSource:0}: Error finding container a36cb67969cb777b2ba514204a8864fb852d71e1ddaf4a165ecbfa265230a1fe: Status 404 returned error can't find the container with id a36cb67969cb777b2ba514204a8864fb852d71e1ddaf4a165ecbfa265230a1fe
Jul 29 19:37:49.695721 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:49.695678    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4" event={"ID":"cd7e4b4c-cf34-4858-85f3-190ffe31edc2","Type":"ContainerStarted","Data":"a36cb67969cb777b2ba514204a8864fb852d71e1ddaf4a165ecbfa265230a1fe"}
Jul 29 19:37:59.121266 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.121230    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"]
Jul 29 19:37:59.128154 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.128128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.130734 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.130709    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dmlqf\""
Jul 29 19:37:59.142258 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.142215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"]
Jul 29 19:37:59.192794 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.192758    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd9be6b0-b11b-40d1-a929-aedbd449d373-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-prn6j\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.192975 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.192813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtdg\" (UniqueName: \"kubernetes.io/projected/bd9be6b0-b11b-40d1-a929-aedbd449d373-kube-api-access-rrtdg\") pod \"kuadrant-operator-controller-manager-86878c769f-prn6j\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.293459 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.293417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd9be6b0-b11b-40d1-a929-aedbd449d373-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-prn6j\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.293663 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.293477    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtdg\" (UniqueName: \"kubernetes.io/projected/bd9be6b0-b11b-40d1-a929-aedbd449d373-kube-api-access-rrtdg\") pod \"kuadrant-operator-controller-manager-86878c769f-prn6j\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.293869 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.293849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd9be6b0-b11b-40d1-a929-aedbd449d373-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-prn6j\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.307078 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.307050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtdg\" (UniqueName: \"kubernetes.io/projected/bd9be6b0-b11b-40d1-a929-aedbd449d373-kube-api-access-rrtdg\") pod \"kuadrant-operator-controller-manager-86878c769f-prn6j\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:37:59.442638 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:37:59.442601    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:38:13.190129 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:13.190087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"]
Jul 29 19:38:13.194599 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:38:13.194550    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9be6b0_b11b_40d1_a929_aedbd449d373.slice/crio-6b900141612c91d25810784481219e30312ed88eadf02203c630bd0e572bfbb5 WatchSource:0}: Error finding container 6b900141612c91d25810784481219e30312ed88eadf02203c630bd0e572bfbb5: Status 404 returned error can't find the container with id 6b900141612c91d25810784481219e30312ed88eadf02203c630bd0e572bfbb5
Jul 29 19:38:13.821008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:13.820921    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4" event={"ID":"cd7e4b4c-cf34-4858-85f3-190ffe31edc2","Type":"ContainerStarted","Data":"b512385557f886f4d87f212728926a2bb0f2abf97c24e0377534d32eb894affa"}
Jul 29 19:38:13.822207 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:13.822180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j" event={"ID":"bd9be6b0-b11b-40d1-a929-aedbd449d373","Type":"ContainerStarted","Data":"6b900141612c91d25810784481219e30312ed88eadf02203c630bd0e572bfbb5"}
Jul 29 19:38:13.842857 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:13.842811    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-x29j4" podStartSLOduration=1.759887193 podStartE2EDuration="25.842796056s" totalImagesPullingTime="24.082908863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:38:13.840935979 +0000 UTC m=+499.799430875" watchObservedRunningTime="2026-07-29 19:38:13.842796056 +0000 UTC m=+499.801290949"
Jul 29 19:38:19.853352 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:19.853315    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j" event={"ID":"bd9be6b0-b11b-40d1-a929-aedbd449d373","Type":"ContainerStarted","Data":"25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea"}
Jul 29 19:38:19.853764 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:19.853426    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:38:19.874785 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:19.874736    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j" podStartSLOduration=14.730424817 podStartE2EDuration="20.874721714s" totalImagesPullingTime="6.144296897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:38:19.87279254 +0000 UTC m=+505.831287453" watchObservedRunningTime="2026-07-29 19:38:19.874721714 +0000 UTC m=+505.833216608"
Jul 29 19:38:30.860176 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:30.860138    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:38:53.672354 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.672272    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-56z2h"]
Jul 29 19:38:53.678726 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.678699    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:38:53.680643 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.680624    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ldhdt\""
Jul 29 19:38:53.686407 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.686011    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-56z2h"]
Jul 29 19:38:53.775414 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.775373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nsv6\" (UniqueName: \"kubernetes.io/projected/87b95075-be2f-4549-84ca-c796336c7332-kube-api-access-2nsv6\") pod \"authorino-75d556bd77-56z2h\" (UID: \"87b95075-be2f-4549-84ca-c796336c7332\") " pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:38:53.876121 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.876085    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsv6\" (UniqueName: \"kubernetes.io/projected/87b95075-be2f-4549-84ca-c796336c7332-kube-api-access-2nsv6\") pod \"authorino-75d556bd77-56z2h\" (UID: \"87b95075-be2f-4549-84ca-c796336c7332\") " pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:38:53.885533 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.885508    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nsv6\" (UniqueName: \"kubernetes.io/projected/87b95075-be2f-4549-84ca-c796336c7332-kube-api-access-2nsv6\") pod \"authorino-75d556bd77-56z2h\" (UID: \"87b95075-be2f-4549-84ca-c796336c7332\") " pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:38:53.988932 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:53.988894    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:38:54.127114 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:54.127083    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-56z2h"]
Jul 29 19:38:54.130231 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:38:54.130204    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87b95075_be2f_4549_84ca_c796336c7332.slice/crio-f23084baa51d1c6f648aa3173a2359a8329779a5e6f4caed9727031cdbbb30db WatchSource:0}: Error finding container f23084baa51d1c6f648aa3173a2359a8329779a5e6f4caed9727031cdbbb30db: Status 404 returned error can't find the container with id f23084baa51d1c6f648aa3173a2359a8329779a5e6f4caed9727031cdbbb30db
Jul 29 19:38:55.006989 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:55.006949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-56z2h" event={"ID":"87b95075-be2f-4549-84ca-c796336c7332","Type":"ContainerStarted","Data":"f23084baa51d1c6f648aa3173a2359a8329779a5e6f4caed9727031cdbbb30db"}
Jul 29 19:38:58.024133 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:58.024092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-56z2h" event={"ID":"87b95075-be2f-4549-84ca-c796336c7332","Type":"ContainerStarted","Data":"ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22"}
Jul 29 19:38:58.040557 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:38:58.040508    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-56z2h" podStartSLOduration=2.113232502 podStartE2EDuration="5.040493004s" totalImagesPullingTime="2.927260502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:38:58.039097145 +0000 UTC m=+543.997592223" watchObservedRunningTime="2026-07-29 19:38:58.040493004 +0000 UTC m=+543.998987897"
Jul 29 19:39:28.826290 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:28.826252    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-56z2h"]
Jul 29 19:39:28.826813 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:28.826449    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-56z2h" podUID="87b95075-be2f-4549-84ca-c796336c7332" containerName="authorino" containerID="cri-o://ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22" gracePeriod=30
Jul 29 19:39:29.087310 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.087245    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:39:29.154149 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.154110    2555 generic.go:356] "Generic (PLEG): container finished" podID="87b95075-be2f-4549-84ca-c796336c7332" containerID="ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22" exitCode=0
Jul 29 19:39:29.154149 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.154152    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-56z2h"
Jul 29 19:39:29.154342 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.154186    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-56z2h" event={"ID":"87b95075-be2f-4549-84ca-c796336c7332","Type":"ContainerDied","Data":"ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22"}
Jul 29 19:39:29.154342 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.154231    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-56z2h" event={"ID":"87b95075-be2f-4549-84ca-c796336c7332","Type":"ContainerDied","Data":"f23084baa51d1c6f648aa3173a2359a8329779a5e6f4caed9727031cdbbb30db"}
Jul 29 19:39:29.154342 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.154250    2555 scope.go:117] "RemoveContainer" containerID="ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22"
Jul 29 19:39:29.163949 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.163921    2555 scope.go:117] "RemoveContainer" containerID="ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22"
Jul 29 19:39:29.164288 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:39:29.164261    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22\": container with ID starting with ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22 not found: ID does not exist" containerID="ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22"
Jul 29 19:39:29.164367 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.164315    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22"} err="failed to get container status \"ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22\": rpc error: code = NotFound desc = could not find container \"ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22\": container with ID starting with ea3de4deebb359005666a494a2fcbfe1fabb117527f46fa16f89e4842dc65f22 not found: ID does not exist"
Jul 29 19:39:29.191485 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.191320    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nsv6\" (UniqueName: \"kubernetes.io/projected/87b95075-be2f-4549-84ca-c796336c7332-kube-api-access-2nsv6\") pod \"87b95075-be2f-4549-84ca-c796336c7332\" (UID: \"87b95075-be2f-4549-84ca-c796336c7332\") "
Jul 29 19:39:29.193490 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.193456    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b95075-be2f-4549-84ca-c796336c7332-kube-api-access-2nsv6" (OuterVolumeSpecName: "kube-api-access-2nsv6") pod "87b95075-be2f-4549-84ca-c796336c7332" (UID: "87b95075-be2f-4549-84ca-c796336c7332"). InnerVolumeSpecName "kube-api-access-2nsv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:39:29.292914 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.292877    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nsv6\" (UniqueName: \"kubernetes.io/projected/87b95075-be2f-4549-84ca-c796336c7332-kube-api-access-2nsv6\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:39:29.480443 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.480405    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-56z2h"]
Jul 29 19:39:29.489728 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:29.489706    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-56z2h"]
Jul 29 19:39:30.658640 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:30.658597    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b95075-be2f-4549-84ca-c796336c7332" path="/var/lib/kubelet/pods/87b95075-be2f-4549-84ca-c796336c7332/volumes"
Jul 29 19:39:33.457794 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.457755    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58b6468fbc-xwhlg"]
Jul 29 19:39:33.458157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.458130    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b95075-be2f-4549-84ca-c796336c7332" containerName="authorino"
Jul 29 19:39:33.458157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.458142    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b95075-be2f-4549-84ca-c796336c7332" containerName="authorino"
Jul 29 19:39:33.458232 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.458210    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b95075-be2f-4549-84ca-c796336c7332" containerName="authorino"
Jul 29 19:39:33.463268 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.463250    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:33.465746 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.465724    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 19:39:33.465866 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.465758    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65bn9\""
Jul 29 19:39:33.466252 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.466230    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 19:39:33.472368 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.472345    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58b6468fbc-xwhlg"]
Jul 29 19:39:33.532379 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.532347    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcsr\" (UniqueName: \"kubernetes.io/projected/6227ad27-9591-4f7f-9ebb-a005e3bafd19-kube-api-access-8kcsr\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:33.532565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.532405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:33.629373 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.629340    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c74855f4-79p7k"]
Jul 29 19:39:33.632181 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.632161    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.632839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.632800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcsr\" (UniqueName: \"kubernetes.io/projected/6227ad27-9591-4f7f-9ebb-a005e3bafd19-kube-api-access-8kcsr\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:33.632980 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.632860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:33.632980 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:39:33.632963    2555 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 29 19:39:33.633087 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:39:33.633033    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert podName:6227ad27-9591-4f7f-9ebb-a005e3bafd19 nodeName:}" failed. No retries permitted until 2026-07-29 19:39:34.133014289 +0000 UTC m=+580.091509165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert") pod "maas-controller-58b6468fbc-xwhlg" (UID: "6227ad27-9591-4f7f-9ebb-a005e3bafd19") : secret "maas-controller-webhook-cert" not found
Jul 29 19:39:33.643107 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.643076    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c74855f4-79p7k"]
Jul 29 19:39:33.646230 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.646206    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcsr\" (UniqueName: \"kubernetes.io/projected/6227ad27-9591-4f7f-9ebb-a005e3bafd19-kube-api-access-8kcsr\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:33.734372 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.734270    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d5aeb9e-770e-408d-a999-634d74813e3a-webhook-cert\") pod \"maas-controller-55c74855f4-79p7k\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") " pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.734372 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.734322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvjv\" (UniqueName: \"kubernetes.io/projected/4d5aeb9e-770e-408d-a999-634d74813e3a-kube-api-access-dgvjv\") pod \"maas-controller-55c74855f4-79p7k\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") " pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.835379 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.835342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d5aeb9e-770e-408d-a999-634d74813e3a-webhook-cert\") pod \"maas-controller-55c74855f4-79p7k\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") " pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.835379 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.835380    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvjv\" (UniqueName: \"kubernetes.io/projected/4d5aeb9e-770e-408d-a999-634d74813e3a-kube-api-access-dgvjv\") pod \"maas-controller-55c74855f4-79p7k\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") " pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.837779 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.837757    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d5aeb9e-770e-408d-a999-634d74813e3a-webhook-cert\") pod \"maas-controller-55c74855f4-79p7k\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") " pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.844723 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.844700    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvjv\" (UniqueName: \"kubernetes.io/projected/4d5aeb9e-770e-408d-a999-634d74813e3a-kube-api-access-dgvjv\") pod \"maas-controller-55c74855f4-79p7k\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") " pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:33.943419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:33.943378    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:34.077426 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:34.077371    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c74855f4-79p7k"]
Jul 29 19:39:34.079735 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:39:34.079706    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5aeb9e_770e_408d_a999_634d74813e3a.slice/crio-41bcf8ce03b53f7431672ac0bdb1fc56c6d460433b5eb76e6f68626674811bb3 WatchSource:0}: Error finding container 41bcf8ce03b53f7431672ac0bdb1fc56c6d460433b5eb76e6f68626674811bb3: Status 404 returned error can't find the container with id 41bcf8ce03b53f7431672ac0bdb1fc56c6d460433b5eb76e6f68626674811bb3
Jul 29 19:39:34.137856 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:34.137821    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:34.144046 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:34.141454    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert\") pod \"maas-controller-58b6468fbc-xwhlg\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") " pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:34.178395 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:34.178357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-79p7k" event={"ID":"4d5aeb9e-770e-408d-a999-634d74813e3a","Type":"ContainerStarted","Data":"41bcf8ce03b53f7431672ac0bdb1fc56c6d460433b5eb76e6f68626674811bb3"}
Jul 29 19:39:34.374946 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:34.374856    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:34.509444 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:34.509415    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58b6468fbc-xwhlg"]
Jul 29 19:39:34.511769 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:39:34.511744    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6227ad27_9591_4f7f_9ebb_a005e3bafd19.slice/crio-e99802e84f3b61ef220347abda8ed011b195de816ffbde86e2dc7f4bd6f33c69 WatchSource:0}: Error finding container e99802e84f3b61ef220347abda8ed011b195de816ffbde86e2dc7f4bd6f33c69: Status 404 returned error can't find the container with id e99802e84f3b61ef220347abda8ed011b195de816ffbde86e2dc7f4bd6f33c69
Jul 29 19:39:35.187568 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:35.187468    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b6468fbc-xwhlg" event={"ID":"6227ad27-9591-4f7f-9ebb-a005e3bafd19","Type":"ContainerStarted","Data":"e99802e84f3b61ef220347abda8ed011b195de816ffbde86e2dc7f4bd6f33c69"}
Jul 29 19:39:37.198488 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:37.198451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-79p7k" event={"ID":"4d5aeb9e-770e-408d-a999-634d74813e3a","Type":"ContainerStarted","Data":"dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe"}
Jul 29 19:39:37.198942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:37.198571    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:37.200130 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:37.200104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b6468fbc-xwhlg" event={"ID":"6227ad27-9591-4f7f-9ebb-a005e3bafd19","Type":"ContainerStarted","Data":"ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d"}
Jul 29 19:39:37.200246 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:37.200182    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:37.218253 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:37.218206    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55c74855f4-79p7k" podStartSLOduration=1.7165600909999998 podStartE2EDuration="4.218192743s" totalImagesPullingTime="2.501632652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:39:37.216128066 +0000 UTC m=+583.174622985" watchObservedRunningTime="2026-07-29 19:39:37.218192743 +0000 UTC m=+583.176687629"
Jul 29 19:39:37.234897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:37.234847    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58b6468fbc-xwhlg" podStartSLOduration=2.156133166 podStartE2EDuration="4.234829476s" totalImagesPullingTime="2.07869631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:39:37.233642838 +0000 UTC m=+583.192137756" watchObservedRunningTime="2026-07-29 19:39:37.234829476 +0000 UTC m=+583.193324370"
Jul 29 19:39:48.206645 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.206612    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:48.207134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.206697    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:39:48.282658 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.282617    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58b6468fbc-xwhlg"]
Jul 29 19:39:48.282891 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.282862    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58b6468fbc-xwhlg" podUID="6227ad27-9591-4f7f-9ebb-a005e3bafd19" containerName="manager" containerID="cri-o://ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d" gracePeriod=30
Jul 29 19:39:48.534405 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.534381    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:48.670682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.670647    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kcsr\" (UniqueName: \"kubernetes.io/projected/6227ad27-9591-4f7f-9ebb-a005e3bafd19-kube-api-access-8kcsr\") pod \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") "
Jul 29 19:39:48.670833 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.670740    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert\") pod \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\" (UID: \"6227ad27-9591-4f7f-9ebb-a005e3bafd19\") "
Jul 29 19:39:48.673044 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.673011    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6227ad27-9591-4f7f-9ebb-a005e3bafd19-kube-api-access-8kcsr" (OuterVolumeSpecName: "kube-api-access-8kcsr") pod "6227ad27-9591-4f7f-9ebb-a005e3bafd19" (UID: "6227ad27-9591-4f7f-9ebb-a005e3bafd19"). InnerVolumeSpecName "kube-api-access-8kcsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:39:48.673161 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.673011    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6227ad27-9591-4f7f-9ebb-a005e3bafd19" (UID: "6227ad27-9591-4f7f-9ebb-a005e3bafd19"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:39:48.771903 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.771823    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6227ad27-9591-4f7f-9ebb-a005e3bafd19-webhook-cert\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:39:48.771903 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:48.771852    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcsr\" (UniqueName: \"kubernetes.io/projected/6227ad27-9591-4f7f-9ebb-a005e3bafd19-kube-api-access-8kcsr\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:39:49.258737 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.258701    2555 generic.go:356] "Generic (PLEG): container finished" podID="6227ad27-9591-4f7f-9ebb-a005e3bafd19" containerID="ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d" exitCode=0
Jul 29 19:39:49.259120 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.258754    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58b6468fbc-xwhlg"
Jul 29 19:39:49.259120 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.258785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b6468fbc-xwhlg" event={"ID":"6227ad27-9591-4f7f-9ebb-a005e3bafd19","Type":"ContainerDied","Data":"ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d"}
Jul 29 19:39:49.259120 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.258845    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58b6468fbc-xwhlg" event={"ID":"6227ad27-9591-4f7f-9ebb-a005e3bafd19","Type":"ContainerDied","Data":"e99802e84f3b61ef220347abda8ed011b195de816ffbde86e2dc7f4bd6f33c69"}
Jul 29 19:39:49.259120 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.258877    2555 scope.go:117] "RemoveContainer" containerID="ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d"
Jul 29 19:39:49.267870 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.267852    2555 scope.go:117] "RemoveContainer" containerID="ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d"
Jul 29 19:39:49.268134 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:39:49.268113    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d\": container with ID starting with ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d not found: ID does not exist" containerID="ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d"
Jul 29 19:39:49.268232 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.268141    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d"} err="failed to get container status \"ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d\": rpc error: code = NotFound desc = could not find container \"ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d\": container with ID starting with ccdb31d6e23e4d7bea6b4753f12ed554bbfa0461f4ce79f3561b9ae85fb15d1d not found: ID does not exist"
Jul 29 19:39:49.280283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.280261    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58b6468fbc-xwhlg"]
Jul 29 19:39:49.286506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:49.286483    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58b6468fbc-xwhlg"]
Jul 29 19:39:50.658054 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:50.658023    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6227ad27-9591-4f7f-9ebb-a005e3bafd19" path="/var/lib/kubelet/pods/6227ad27-9591-4f7f-9ebb-a005e3bafd19/volumes"
Jul 29 19:39:54.501387 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:54.501358    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:39:54.505524 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:39:54.505504    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:40:09.004948 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.004915    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"]
Jul 29 19:40:09.005426 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.005303    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6227ad27-9591-4f7f-9ebb-a005e3bafd19" containerName="manager"
Jul 29 19:40:09.005426 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.005323    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6227ad27-9591-4f7f-9ebb-a005e3bafd19" containerName="manager"
Jul 29 19:40:09.005426 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.005419    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6227ad27-9591-4f7f-9ebb-a005e3bafd19" containerName="manager"
Jul 29 19:40:09.013778 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.013750    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.015692 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.015659    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 29 19:40:09.015808 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.015725    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 19:40:09.015872 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.015851    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 19:40:09.030861 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.030836    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"]
Jul 29 19:40:09.048186 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.048154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265zx\" (UniqueName: \"kubernetes.io/projected/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-kube-api-access-265zx\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.048323 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.048202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-maas-api-tls\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.048409 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.048356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-openshift-service-ca\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.149485 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.149447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-openshift-service-ca\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.149485 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.149487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265zx\" (UniqueName: \"kubernetes.io/projected/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-kube-api-access-265zx\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.149700 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.149513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-maas-api-tls\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.150092 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.150071    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-openshift-service-ca\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.152154 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.152129    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-maas-api-tls\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.162627 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.159801    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-265zx\" (UniqueName: \"kubernetes.io/projected/7055208f-2e5b-4c9f-8a57-ac1133fdaffe-kube-api-access-265zx\") pod \"maas-api-776f87874c-b7jnb\" (UID: \"7055208f-2e5b-4c9f-8a57-ac1133fdaffe\") " pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.324211 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.324119    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:09.466458 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.466425    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"]
Jul 29 19:40:09.470721 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:40:09.470693    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7055208f_2e5b_4c9f_8a57_ac1133fdaffe.slice/crio-08753130bbf4e5596b0741eae4fe003b4cf7ab8267b58b10a9f2d8854ed3a29a WatchSource:0}: Error finding container 08753130bbf4e5596b0741eae4fe003b4cf7ab8267b58b10a9f2d8854ed3a29a: Status 404 returned error can't find the container with id 08753130bbf4e5596b0741eae4fe003b4cf7ab8267b58b10a9f2d8854ed3a29a
Jul 29 19:40:09.963612 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.963557    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hwgbd"]
Jul 29 19:40:09.968893 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.968870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:09.971746 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.971723    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 19:40:09.971883 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.971722    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t4xgq\""
Jul 29 19:40:09.988799 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:09.988767    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hwgbd"]
Jul 29 19:40:10.058826 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.058793    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d222932-eac3-4e73-954b-af06b118b1d5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hwgbd\" (UID: \"9d222932-eac3-4e73-954b-af06b118b1d5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.059216 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.058842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnt4\" (UniqueName: \"kubernetes.io/projected/9d222932-eac3-4e73-954b-af06b118b1d5-kube-api-access-mlnt4\") pod \"payload-processing-5cc6cd444b-hwgbd\" (UID: \"9d222932-eac3-4e73-954b-af06b118b1d5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.159828 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.159788    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnt4\" (UniqueName: \"kubernetes.io/projected/9d222932-eac3-4e73-954b-af06b118b1d5-kube-api-access-mlnt4\") pod \"payload-processing-5cc6cd444b-hwgbd\" (UID: \"9d222932-eac3-4e73-954b-af06b118b1d5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.160008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.159911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d222932-eac3-4e73-954b-af06b118b1d5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hwgbd\" (UID: \"9d222932-eac3-4e73-954b-af06b118b1d5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.160772 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.160746    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d222932-eac3-4e73-954b-af06b118b1d5-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hwgbd\" (UID: \"9d222932-eac3-4e73-954b-af06b118b1d5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.169673 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.169643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnt4\" (UniqueName: \"kubernetes.io/projected/9d222932-eac3-4e73-954b-af06b118b1d5-kube-api-access-mlnt4\") pod \"payload-processing-5cc6cd444b-hwgbd\" (UID: \"9d222932-eac3-4e73-954b-af06b118b1d5\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.281499 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.281415    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:10.354571 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.354537    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb" event={"ID":"7055208f-2e5b-4c9f-8a57-ac1133fdaffe","Type":"ContainerStarted","Data":"08753130bbf4e5596b0741eae4fe003b4cf7ab8267b58b10a9f2d8854ed3a29a"}
Jul 29 19:40:10.419929 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:10.419905    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hwgbd"]
Jul 29 19:40:10.422087 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:40:10.422050    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d222932_eac3_4e73_954b_af06b118b1d5.slice/crio-cd46510cbc22758d43d37640f4a9a3537ea8ebb560e4451e306f7e96af1cd4bd WatchSource:0}: Error finding container cd46510cbc22758d43d37640f4a9a3537ea8ebb560e4451e306f7e96af1cd4bd: Status 404 returned error can't find the container with id cd46510cbc22758d43d37640f4a9a3537ea8ebb560e4451e306f7e96af1cd4bd
Jul 29 19:40:11.362999 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:11.362943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd" event={"ID":"9d222932-eac3-4e73-954b-af06b118b1d5","Type":"ContainerStarted","Data":"cd46510cbc22758d43d37640f4a9a3537ea8ebb560e4451e306f7e96af1cd4bd"}
Jul 29 19:40:12.370121 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:12.370085    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb" event={"ID":"7055208f-2e5b-4c9f-8a57-ac1133fdaffe","Type":"ContainerStarted","Data":"c90f1441fb134ba0733199371728a5643424fed72b7ddc50222740e36f250ccf"}
Jul 29 19:40:12.370612 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:12.370188    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:12.388463 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:12.388402    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb" podStartSLOduration=2.170461251 podStartE2EDuration="4.388378972s" totalImagesPullingTime="2.217917721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:12.386871768 +0000 UTC m=+618.345366701" watchObservedRunningTime="2026-07-29 19:40:12.388378972 +0000 UTC m=+618.346873867"
Jul 29 19:40:13.375746 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:13.375703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd" event={"ID":"9d222932-eac3-4e73-954b-af06b118b1d5","Type":"ContainerStarted","Data":"74a388b7e4ba684c0be8408aa87e8a011f15490a56cfae9209d5a1921194b24e"}
Jul 29 19:40:13.376174 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:13.375861    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:13.400342 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:13.400275    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd" podStartSLOduration=2.002805126 podStartE2EDuration="4.400261406s" totalImagesPullingTime="2.39745628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:13.398216785 +0000 UTC m=+619.356711698" watchObservedRunningTime="2026-07-29 19:40:13.400261406 +0000 UTC m=+619.358756300"
Jul 29 19:40:18.381360 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:18.381281    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-776f87874c-b7jnb"
Jul 29 19:40:24.382688 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:24.382655    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hwgbd"
Jul 29 19:40:41.793317 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.793263    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"]
Jul 29 19:40:41.796667 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.796650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:41.798972 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.798941    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8fhrs\""
Jul 29 19:40:41.799107 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.798949    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 29 19:40:41.799107 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.798985    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 19:40:41.799107 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.799007    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 19:40:41.806251 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.806225    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"]
Jul 29 19:40:41.948467 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.948429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:41.948682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.948496    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:41.948682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.948523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d688b1d-dfad-4a6e-a64d-d71e971859c3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:41.948682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.948567    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:41.948682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.948618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr72b\" (UniqueName: \"kubernetes.io/projected/1d688b1d-dfad-4a6e-a64d-d71e971859c3-kube-api-access-vr72b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:41.948682 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:41.948663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049412 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049412 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049380    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049412 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d688b1d-dfad-4a6e-a64d-d71e971859c3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049412 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049722 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049434    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr72b\" (UniqueName: \"kubernetes.io/projected/1d688b1d-dfad-4a6e-a64d-d71e971859c3-kube-api-access-vr72b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049722 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049825 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049800    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049864 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.049921 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.049904    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.051960 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.051933    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d688b1d-dfad-4a6e-a64d-d71e971859c3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.052126 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.052110    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d688b1d-dfad-4a6e-a64d-d71e971859c3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.062106 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.062085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr72b\" (UniqueName: \"kubernetes.io/projected/1d688b1d-dfad-4a6e-a64d-d71e971859c3-kube-api-access-vr72b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9zhpn\" (UID: \"1d688b1d-dfad-4a6e-a64d-d71e971859c3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.108600 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.108537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:42.253027 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.253001    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"]
Jul 29 19:40:42.255716 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:40:42.255688    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d688b1d_dfad_4a6e_a64d_d71e971859c3.slice/crio-940cb00cc1c68d8473241155916845505dc4240fe612624203cba08cd840296c WatchSource:0}: Error finding container 940cb00cc1c68d8473241155916845505dc4240fe612624203cba08cd840296c: Status 404 returned error can't find the container with id 940cb00cc1c68d8473241155916845505dc4240fe612624203cba08cd840296c
Jul 29 19:40:42.257113 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.257091    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:40:42.506534 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:42.506493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn" event={"ID":"1d688b1d-dfad-4a6e-a64d-d71e971859c3","Type":"ContainerStarted","Data":"940cb00cc1c68d8473241155916845505dc4240fe612624203cba08cd840296c"}
Jul 29 19:40:43.526438 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:43.526403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn" event={"ID":"1d688b1d-dfad-4a6e-a64d-d71e971859c3","Type":"ContainerStarted","Data":"5d7e70957ef4e250a32cdb1ca50e34d897195fab34502a743f92be87c397dfc7"}
Jul 29 19:40:43.544485 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:43.544435    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn" podStartSLOduration=1.377640664 podStartE2EDuration="2.544422443s" totalImagesPullingTime="1.166781779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:40:43.543329297 +0000 UTC m=+649.501824206" watchObservedRunningTime="2026-07-29 19:40:43.544422443 +0000 UTC m=+649.502917337"
Jul 29 19:40:52.109180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:52.109128    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:52.109180 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:52.109201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:52.121567 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:52.121542    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:40:52.577571 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:40:52.577542    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9zhpn"
Jul 29 19:41:46.178398 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.178307    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"]
Jul 29 19:41:46.182045 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.182019    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.183871 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.183848    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 19:41:46.191559 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.191009    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"]
Jul 29 19:41:46.220603 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.220549    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.220603 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.220604    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45219a0f-5f75-4ca0-8c62-6b4bdd297151-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.220801 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.220657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.220801 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.220673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.220801 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.220697    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w2v5\" (UniqueName: \"kubernetes.io/projected/45219a0f-5f75-4ca0-8c62-6b4bdd297151-kube-api-access-4w2v5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.220801 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.220724    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321326 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321326 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45219a0f-5f75-4ca0-8c62-6b4bdd297151-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321624 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321624 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321624 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w2v5\" (UniqueName: \"kubernetes.io/projected/45219a0f-5f75-4ca0-8c62-6b4bdd297151-kube-api-access-4w2v5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321624 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321836 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321807    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321896 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.321950 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.321923    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.324034 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.324000    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45219a0f-5f75-4ca0-8c62-6b4bdd297151-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.324156 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.324139    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45219a0f-5f75-4ca0-8c62-6b4bdd297151-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.330282 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.330259    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w2v5\" (UniqueName: \"kubernetes.io/projected/45219a0f-5f75-4ca0-8c62-6b4bdd297151-kube-api-access-4w2v5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4\" (UID: \"45219a0f-5f75-4ca0-8c62-6b4bdd297151\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.494532 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.494482    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:46.628565 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.628455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"]
Jul 29 19:41:46.631280 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:41:46.631252    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45219a0f_5f75_4ca0_8c62_6b4bdd297151.slice/crio-db501991a926df0a9e658781a4dd75aca353b3d4619f0d78f36751cda2f7a699 WatchSource:0}: Error finding container db501991a926df0a9e658781a4dd75aca353b3d4619f0d78f36751cda2f7a699: Status 404 returned error can't find the container with id db501991a926df0a9e658781a4dd75aca353b3d4619f0d78f36751cda2f7a699
Jul 29 19:41:46.812108 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:46.812023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4" event={"ID":"45219a0f-5f75-4ca0-8c62-6b4bdd297151","Type":"ContainerStarted","Data":"db501991a926df0a9e658781a4dd75aca353b3d4619f0d78f36751cda2f7a699"}
Jul 29 19:41:47.818085 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:47.818044    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4" event={"ID":"45219a0f-5f75-4ca0-8c62-6b4bdd297151","Type":"ContainerStarted","Data":"9246f530a5f3159a1b1fd3ce86db76df45b5906a91336238eba0d2cbf12fab81"}
Jul 29 19:41:47.837409 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:47.837361    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4" podStartSLOduration=1.627611353 podStartE2EDuration="1.837343881s" totalImagesPullingTime="209.732528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:41:47.835889542 +0000 UTC m=+713.794384437" watchObservedRunningTime="2026-07-29 19:41:47.837343881 +0000 UTC m=+713.795838772"
Jul 29 19:41:56.494654 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:56.494616    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:56.495053 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:56.494873    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:56.507769 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:56.507737    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:41:56.873283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:41:56.873191    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4"
Jul 29 19:43:51.434797 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.434759    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c74855f4-79p7k"]
Jul 29 19:43:51.435238 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.434980    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55c74855f4-79p7k" podUID="4d5aeb9e-770e-408d-a999-634d74813e3a" containerName="manager" containerID="cri-o://dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe" gracePeriod=30
Jul 29 19:43:51.686480 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.686425    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:43:51.746933 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.746904    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgvjv\" (UniqueName: \"kubernetes.io/projected/4d5aeb9e-770e-408d-a999-634d74813e3a-kube-api-access-dgvjv\") pod \"4d5aeb9e-770e-408d-a999-634d74813e3a\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") "
Jul 29 19:43:51.747104 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.746944    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d5aeb9e-770e-408d-a999-634d74813e3a-webhook-cert\") pod \"4d5aeb9e-770e-408d-a999-634d74813e3a\" (UID: \"4d5aeb9e-770e-408d-a999-634d74813e3a\") "
Jul 29 19:43:51.749170 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.749139    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5aeb9e-770e-408d-a999-634d74813e3a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4d5aeb9e-770e-408d-a999-634d74813e3a" (UID: "4d5aeb9e-770e-408d-a999-634d74813e3a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:43:51.749322 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.749210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5aeb9e-770e-408d-a999-634d74813e3a-kube-api-access-dgvjv" (OuterVolumeSpecName: "kube-api-access-dgvjv") pod "4d5aeb9e-770e-408d-a999-634d74813e3a" (UID: "4d5aeb9e-770e-408d-a999-634d74813e3a"). InnerVolumeSpecName "kube-api-access-dgvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:43:51.847874 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.847840    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvjv\" (UniqueName: \"kubernetes.io/projected/4d5aeb9e-770e-408d-a999-634d74813e3a-kube-api-access-dgvjv\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:43:51.847874 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:51.847872    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d5aeb9e-770e-408d-a999-634d74813e3a-webhook-cert\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:43:52.394915 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.394876    2555 generic.go:356] "Generic (PLEG): container finished" podID="4d5aeb9e-770e-408d-a999-634d74813e3a" containerID="dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe" exitCode=0
Jul 29 19:43:52.394915 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.394917    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-79p7k"
Jul 29 19:43:52.395163 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.394929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-79p7k" event={"ID":"4d5aeb9e-770e-408d-a999-634d74813e3a","Type":"ContainerDied","Data":"dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe"}
Jul 29 19:43:52.395163 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.394992    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-79p7k" event={"ID":"4d5aeb9e-770e-408d-a999-634d74813e3a","Type":"ContainerDied","Data":"41bcf8ce03b53f7431672ac0bdb1fc56c6d460433b5eb76e6f68626674811bb3"}
Jul 29 19:43:52.395163 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.395022    2555 scope.go:117] "RemoveContainer" containerID="dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe"
Jul 29 19:43:52.403946 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.403927    2555 scope.go:117] "RemoveContainer" containerID="dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe"
Jul 29 19:43:52.404202 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:43:52.404184    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe\": container with ID starting with dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe not found: ID does not exist" containerID="dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe"
Jul 29 19:43:52.404246 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.404210    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe"} err="failed to get container status \"dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe\": rpc error: code = NotFound desc = could not find container \"dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe\": container with ID starting with dfd6d5d579e3b7e4f5929b8274e2fd742212515ad86f8c0009415a14d4c206fe not found: ID does not exist"
Jul 29 19:43:52.416800 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.416776    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c74855f4-79p7k"]
Jul 29 19:43:52.421030 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.421008    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55c74855f4-79p7k"]
Jul 29 19:43:52.661764 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:43:52.659305    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5aeb9e-770e-408d-a999-634d74813e3a" path="/var/lib/kubelet/pods/4d5aeb9e-770e-408d-a999-634d74813e3a/volumes"
Jul 29 19:44:53.219185 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.219094    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c74855f4-cbhpq"]
Jul 29 19:44:53.219679 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.219480    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5aeb9e-770e-408d-a999-634d74813e3a" containerName="manager"
Jul 29 19:44:53.219679 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.219498    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5aeb9e-770e-408d-a999-634d74813e3a" containerName="manager"
Jul 29 19:44:53.219679 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.219552    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5aeb9e-770e-408d-a999-634d74813e3a" containerName="manager"
Jul 29 19:44:53.222777 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.222756    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.225350 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.225327    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 19:44:53.225450 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.225363    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65bn9\""
Jul 29 19:44:53.225450 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.225363    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 19:44:53.232994 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.232971    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c74855f4-cbhpq"]
Jul 29 19:44:53.291272 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.291246    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a49495-e3fd-409f-b5f4-080d4d86aba8-webhook-cert\") pod \"maas-controller-55c74855f4-cbhpq\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") " pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.291400 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.291316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwkf\" (UniqueName: \"kubernetes.io/projected/67a49495-e3fd-409f-b5f4-080d4d86aba8-kube-api-access-zrwkf\") pod \"maas-controller-55c74855f4-cbhpq\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") " pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.392428 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.392396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwkf\" (UniqueName: \"kubernetes.io/projected/67a49495-e3fd-409f-b5f4-080d4d86aba8-kube-api-access-zrwkf\") pod \"maas-controller-55c74855f4-cbhpq\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") " pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.392613 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.392460    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a49495-e3fd-409f-b5f4-080d4d86aba8-webhook-cert\") pod \"maas-controller-55c74855f4-cbhpq\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") " pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.394986 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.394963    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a49495-e3fd-409f-b5f4-080d4d86aba8-webhook-cert\") pod \"maas-controller-55c74855f4-cbhpq\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") " pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.400639 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.400614    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwkf\" (UniqueName: \"kubernetes.io/projected/67a49495-e3fd-409f-b5f4-080d4d86aba8-kube-api-access-zrwkf\") pod \"maas-controller-55c74855f4-cbhpq\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") " pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.553527 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.553442    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:53.696133 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:53.696107    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c74855f4-cbhpq"]
Jul 29 19:44:53.698270 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:44:53.698247    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a49495_e3fd_409f_b5f4_080d4d86aba8.slice/crio-03cf8f4055d13c6ddfb9925731f5da389fb72b0ba1e687a65e098d891eae0e6e WatchSource:0}: Error finding container 03cf8f4055d13c6ddfb9925731f5da389fb72b0ba1e687a65e098d891eae0e6e: Status 404 returned error can't find the container with id 03cf8f4055d13c6ddfb9925731f5da389fb72b0ba1e687a65e098d891eae0e6e
Jul 29 19:44:54.532658 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:54.532556    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:44:54.534542 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:54.534523    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:44:54.680068 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:54.680026    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-cbhpq" event={"ID":"67a49495-e3fd-409f-b5f4-080d4d86aba8","Type":"ContainerStarted","Data":"0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651"}
Jul 29 19:44:54.680261 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:54.680093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-cbhpq" event={"ID":"67a49495-e3fd-409f-b5f4-080d4d86aba8","Type":"ContainerStarted","Data":"03cf8f4055d13c6ddfb9925731f5da389fb72b0ba1e687a65e098d891eae0e6e"}
Jul 29 19:44:54.680261 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:54.680198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:44:54.697671 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:44:54.697611    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55c74855f4-cbhpq" podStartSLOduration=1.138730515 podStartE2EDuration="1.69757406s" totalImagesPullingTime="558.843545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:44:54.696375727 +0000 UTC m=+900.654870623" watchObservedRunningTime="2026-07-29 19:44:54.69757406 +0000 UTC m=+900.656068953"
Jul 29 19:45:05.687889 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:05.687854    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:45:10.321784 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.321750    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c74855f4-cbhpq"]
Jul 29 19:45:10.322182 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.321989    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55c74855f4-cbhpq" podUID="67a49495-e3fd-409f-b5f4-080d4d86aba8" containerName="manager" containerID="cri-o://0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651" gracePeriod=30
Jul 29 19:45:10.575190 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.575135    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:45:10.743174 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.743137    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrwkf\" (UniqueName: \"kubernetes.io/projected/67a49495-e3fd-409f-b5f4-080d4d86aba8-kube-api-access-zrwkf\") pod \"67a49495-e3fd-409f-b5f4-080d4d86aba8\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") "
Jul 29 19:45:10.743351 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.743186    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a49495-e3fd-409f-b5f4-080d4d86aba8-webhook-cert\") pod \"67a49495-e3fd-409f-b5f4-080d4d86aba8\" (UID: \"67a49495-e3fd-409f-b5f4-080d4d86aba8\") "
Jul 29 19:45:10.745267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.745236    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a49495-e3fd-409f-b5f4-080d4d86aba8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "67a49495-e3fd-409f-b5f4-080d4d86aba8" (UID: "67a49495-e3fd-409f-b5f4-080d4d86aba8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:45:10.745387 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.745369    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a49495-e3fd-409f-b5f4-080d4d86aba8-kube-api-access-zrwkf" (OuterVolumeSpecName: "kube-api-access-zrwkf") pod "67a49495-e3fd-409f-b5f4-080d4d86aba8" (UID: "67a49495-e3fd-409f-b5f4-080d4d86aba8"). InnerVolumeSpecName "kube-api-access-zrwkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:45:10.761146 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.761113    2555 generic.go:356] "Generic (PLEG): container finished" podID="67a49495-e3fd-409f-b5f4-080d4d86aba8" containerID="0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651" exitCode=0
Jul 29 19:45:10.761267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.761160    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-cbhpq"
Jul 29 19:45:10.761267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.761192    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-cbhpq" event={"ID":"67a49495-e3fd-409f-b5f4-080d4d86aba8","Type":"ContainerDied","Data":"0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651"}
Jul 29 19:45:10.761267 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.761247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-cbhpq" event={"ID":"67a49495-e3fd-409f-b5f4-080d4d86aba8","Type":"ContainerDied","Data":"03cf8f4055d13c6ddfb9925731f5da389fb72b0ba1e687a65e098d891eae0e6e"}
Jul 29 19:45:10.761371 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.761269    2555 scope.go:117] "RemoveContainer" containerID="0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651"
Jul 29 19:45:10.770395 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.770379    2555 scope.go:117] "RemoveContainer" containerID="0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651"
Jul 29 19:45:10.770681 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:45:10.770663    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651\": container with ID starting with 0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651 not found: ID does not exist" containerID="0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651"
Jul 29 19:45:10.770760 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.770687    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651"} err="failed to get container status \"0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651\": rpc error: code = NotFound desc = could not find container \"0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651\": container with ID starting with 0dd83f0eb59f78a7c985305e00ada02937e80cf0e9fff281b787c77d308b1651 not found: ID does not exist"
Jul 29 19:45:10.785000 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.784955    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c74855f4-cbhpq"]
Jul 29 19:45:10.790164 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.790141    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55c74855f4-cbhpq"]
Jul 29 19:45:10.844044 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.843982    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwkf\" (UniqueName: \"kubernetes.io/projected/67a49495-e3fd-409f-b5f4-080d4d86aba8-kube-api-access-zrwkf\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:45:10.844044 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:10.844006    2555 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a49495-e3fd-409f-b5f4-080d4d86aba8-webhook-cert\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:45:11.536161 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.536128    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c74855f4-dmcx7"]
Jul 29 19:45:11.536530 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.536515    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a49495-e3fd-409f-b5f4-080d4d86aba8" containerName="manager"
Jul 29 19:45:11.536530 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.536527    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a49495-e3fd-409f-b5f4-080d4d86aba8" containerName="manager"
Jul 29 19:45:11.536636 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.536601    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a49495-e3fd-409f-b5f4-080d4d86aba8" containerName="manager"
Jul 29 19:45:11.540931 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.540910    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.543348 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.543323    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 19:45:11.543459 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.543314    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-65bn9\""
Jul 29 19:45:11.543459 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.543357    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 19:45:11.551240 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.551215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c74855f4-dmcx7"]
Jul 29 19:45:11.652977 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.652936    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fa5223b-f9d0-41d4-92af-ffa4896de0f9-webhook-cert\") pod \"maas-controller-55c74855f4-dmcx7\" (UID: \"6fa5223b-f9d0-41d4-92af-ffa4896de0f9\") " pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.653153 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.653059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9zd\" (UniqueName: \"kubernetes.io/projected/6fa5223b-f9d0-41d4-92af-ffa4896de0f9-kube-api-access-6h9zd\") pod \"maas-controller-55c74855f4-dmcx7\" (UID: \"6fa5223b-f9d0-41d4-92af-ffa4896de0f9\") " pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.753709 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.753671    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fa5223b-f9d0-41d4-92af-ffa4896de0f9-webhook-cert\") pod \"maas-controller-55c74855f4-dmcx7\" (UID: \"6fa5223b-f9d0-41d4-92af-ffa4896de0f9\") " pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.753925 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.753764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9zd\" (UniqueName: \"kubernetes.io/projected/6fa5223b-f9d0-41d4-92af-ffa4896de0f9-kube-api-access-6h9zd\") pod \"maas-controller-55c74855f4-dmcx7\" (UID: \"6fa5223b-f9d0-41d4-92af-ffa4896de0f9\") " pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.756250 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.756229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6fa5223b-f9d0-41d4-92af-ffa4896de0f9-webhook-cert\") pod \"maas-controller-55c74855f4-dmcx7\" (UID: \"6fa5223b-f9d0-41d4-92af-ffa4896de0f9\") " pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.762783 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.762733    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9zd\" (UniqueName: \"kubernetes.io/projected/6fa5223b-f9d0-41d4-92af-ffa4896de0f9-kube-api-access-6h9zd\") pod \"maas-controller-55c74855f4-dmcx7\" (UID: \"6fa5223b-f9d0-41d4-92af-ffa4896de0f9\") " pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.853942 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.853854    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:11.991780 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:11.991754    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c74855f4-dmcx7"]
Jul 29 19:45:11.994368 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:45:11.994342    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa5223b_f9d0_41d4_92af_ffa4896de0f9.slice/crio-b9a058b58c62adb94aa398e78687fa7d6c6637fa7686cc4366a05e0448f78b41 WatchSource:0}: Error finding container b9a058b58c62adb94aa398e78687fa7d6c6637fa7686cc4366a05e0448f78b41: Status 404 returned error can't find the container with id b9a058b58c62adb94aa398e78687fa7d6c6637fa7686cc4366a05e0448f78b41
Jul 29 19:45:12.660950 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:12.660864    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a49495-e3fd-409f-b5f4-080d4d86aba8" path="/var/lib/kubelet/pods/67a49495-e3fd-409f-b5f4-080d4d86aba8/volumes"
Jul 29 19:45:12.773009 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:12.772975    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-dmcx7" event={"ID":"6fa5223b-f9d0-41d4-92af-ffa4896de0f9","Type":"ContainerStarted","Data":"83a07b75bbb4414dcd8910bf18ae3cdf7d9f838a6ac35fe3beac8bc5b5f83e5a"}
Jul 29 19:45:12.773157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:12.773029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c74855f4-dmcx7" event={"ID":"6fa5223b-f9d0-41d4-92af-ffa4896de0f9","Type":"ContainerStarted","Data":"b9a058b58c62adb94aa398e78687fa7d6c6637fa7686cc4366a05e0448f78b41"}
Jul 29 19:45:12.773157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:12.773115    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:45:12.790550 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:12.790499    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55c74855f4-dmcx7" podStartSLOduration=1.379392731 podStartE2EDuration="1.790484842s" totalImagesPullingTime="411.092111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:45:12.789245005 +0000 UTC m=+918.747739899" watchObservedRunningTime="2026-07-29 19:45:12.790484842 +0000 UTC m=+918.748979737"
Jul 29 19:45:23.780447 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:45:23.780412    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c74855f4-dmcx7"
Jul 29 19:49:54.565600 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:49:54.565505    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:49:54.566922 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:49:54.566889    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:50:06.444478 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:06.444437    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"]
Jul 29 19:50:06.444895 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:06.444705    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j" podUID="bd9be6b0-b11b-40d1-a929-aedbd449d373" containerName="manager" containerID="cri-o://25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea" gracePeriod=10
Jul 29 19:50:08.100023 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.099997    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:50:08.148506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.148468    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd9be6b0-b11b-40d1-a929-aedbd449d373" containerID="25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea" exitCode=0
Jul 29 19:50:08.148697 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.148528    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"
Jul 29 19:50:08.148697 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.148541    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j" event={"ID":"bd9be6b0-b11b-40d1-a929-aedbd449d373","Type":"ContainerDied","Data":"25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea"}
Jul 29 19:50:08.148697 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.148629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j" event={"ID":"bd9be6b0-b11b-40d1-a929-aedbd449d373","Type":"ContainerDied","Data":"6b900141612c91d25810784481219e30312ed88eadf02203c630bd0e572bfbb5"}
Jul 29 19:50:08.148697 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.148664    2555 scope.go:117] "RemoveContainer" containerID="25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea"
Jul 29 19:50:08.158408 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.158391    2555 scope.go:117] "RemoveContainer" containerID="25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea"
Jul 29 19:50:08.158772 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:50:08.158745    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea\": container with ID starting with 25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea not found: ID does not exist" containerID="25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea"
Jul 29 19:50:08.158872 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.158775    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea"} err="failed to get container status \"25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea\": rpc error: code = NotFound desc = could not find container \"25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea\": container with ID starting with 25e5841b2c764414e5a8010c8d0002e2407b4719790d6654edce8bdaf87f86ea not found: ID does not exist"
Jul 29 19:50:08.207014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.206947    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrtdg\" (UniqueName: \"kubernetes.io/projected/bd9be6b0-b11b-40d1-a929-aedbd449d373-kube-api-access-rrtdg\") pod \"bd9be6b0-b11b-40d1-a929-aedbd449d373\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") "
Jul 29 19:50:08.207014 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.207006    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd9be6b0-b11b-40d1-a929-aedbd449d373-extensions-socket-volume\") pod \"bd9be6b0-b11b-40d1-a929-aedbd449d373\" (UID: \"bd9be6b0-b11b-40d1-a929-aedbd449d373\") "
Jul 29 19:50:08.207376 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.207354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9be6b0-b11b-40d1-a929-aedbd449d373-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bd9be6b0-b11b-40d1-a929-aedbd449d373" (UID: "bd9be6b0-b11b-40d1-a929-aedbd449d373"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:50:08.209263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.209237    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9be6b0-b11b-40d1-a929-aedbd449d373-kube-api-access-rrtdg" (OuterVolumeSpecName: "kube-api-access-rrtdg") pod "bd9be6b0-b11b-40d1-a929-aedbd449d373" (UID: "bd9be6b0-b11b-40d1-a929-aedbd449d373"). InnerVolumeSpecName "kube-api-access-rrtdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:50:08.308162 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.308116    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrtdg\" (UniqueName: \"kubernetes.io/projected/bd9be6b0-b11b-40d1-a929-aedbd449d373-kube-api-access-rrtdg\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:50:08.308162 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.308152    2555 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd9be6b0-b11b-40d1-a929-aedbd449d373-extensions-socket-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:50:08.471122 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.471050    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"]
Jul 29 19:50:08.475559 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.475531    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-prn6j"]
Jul 29 19:50:08.658690 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:50:08.658661    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd9be6b0-b11b-40d1-a929-aedbd449d373" path="/var/lib/kubelet/pods/bd9be6b0-b11b-40d1-a929-aedbd449d373/volumes"
Jul 29 19:51:07.585560 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.585523    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"]
Jul 29 19:51:07.586003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.585913    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9be6b0-b11b-40d1-a929-aedbd449d373" containerName="manager"
Jul 29 19:51:07.586003 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.585926    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9be6b0-b11b-40d1-a929-aedbd449d373" containerName="manager"
Jul 29 19:51:07.586094 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.586006    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd9be6b0-b11b-40d1-a929-aedbd449d373" containerName="manager"
Jul 29 19:51:07.589208 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.589191    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.591263 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.591241    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dmlqf\""
Jul 29 19:51:07.602512 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.602489    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"]
Jul 29 19:51:07.741570 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.741528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d46e063-fdfa-463f-8b83-690b5a74c4cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k7n79\" (UID: \"7d46e063-fdfa-463f-8b83-690b5a74c4cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.741771 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.741737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27b75\" (UniqueName: \"kubernetes.io/projected/7d46e063-fdfa-463f-8b83-690b5a74c4cb-kube-api-access-27b75\") pod \"kuadrant-operator-controller-manager-86878c769f-k7n79\" (UID: \"7d46e063-fdfa-463f-8b83-690b5a74c4cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.842998 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.842911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27b75\" (UniqueName: \"kubernetes.io/projected/7d46e063-fdfa-463f-8b83-690b5a74c4cb-kube-api-access-27b75\") pod \"kuadrant-operator-controller-manager-86878c769f-k7n79\" (UID: \"7d46e063-fdfa-463f-8b83-690b5a74c4cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.842998 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.842958    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d46e063-fdfa-463f-8b83-690b5a74c4cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k7n79\" (UID: \"7d46e063-fdfa-463f-8b83-690b5a74c4cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.843316 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.843300    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d46e063-fdfa-463f-8b83-690b5a74c4cb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k7n79\" (UID: \"7d46e063-fdfa-463f-8b83-690b5a74c4cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.855040 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.855010    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27b75\" (UniqueName: \"kubernetes.io/projected/7d46e063-fdfa-463f-8b83-690b5a74c4cb-kube-api-access-27b75\") pod \"kuadrant-operator-controller-manager-86878c769f-k7n79\" (UID: \"7d46e063-fdfa-463f-8b83-690b5a74c4cb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:07.899019 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:07.898984    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:08.035853 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:08.035816    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"]
Jul 29 19:51:08.039496 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:51:08.039459    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d46e063_fdfa_463f_8b83_690b5a74c4cb.slice/crio-d1195207b9a18db61f2f8b0a823aa5b9a63f68be39af89d46a83c3b679db7ae8 WatchSource:0}: Error finding container d1195207b9a18db61f2f8b0a823aa5b9a63f68be39af89d46a83c3b679db7ae8: Status 404 returned error can't find the container with id d1195207b9a18db61f2f8b0a823aa5b9a63f68be39af89d46a83c3b679db7ae8
Jul 29 19:51:08.424388 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:08.424296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79" event={"ID":"7d46e063-fdfa-463f-8b83-690b5a74c4cb","Type":"ContainerStarted","Data":"e1a60decc56571b1a60f578a47093ad862619897abb4b4a7afc6ea73aa210039"}
Jul 29 19:51:08.424388 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:08.424363    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79" event={"ID":"7d46e063-fdfa-463f-8b83-690b5a74c4cb","Type":"ContainerStarted","Data":"d1195207b9a18db61f2f8b0a823aa5b9a63f68be39af89d46a83c3b679db7ae8"}
Jul 29 19:51:08.424564 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:08.424478    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:51:08.449933 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:08.449878    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79" podStartSLOduration=1.4498601629999999 podStartE2EDuration="1.449860163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:51:08.448304066 +0000 UTC m=+1274.406798979" watchObservedRunningTime="2026-07-29 19:51:08.449860163 +0000 UTC m=+1274.408355057"
Jul 29 19:51:19.430528 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:51:19.430496    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7n79"
Jul 29 19:54:01.675157 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.675120    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"]
Jul 29 19:54:01.678662 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.678640    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.680717 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.680695    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fa67c245-kserve-self-signed-certs\""
Jul 29 19:54:01.694000 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.693969    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"]
Jul 29 19:54:01.791771 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.791732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-dshm\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.791771 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.791771    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-model-cache\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.791993 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.791844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e44d6e-1711-4d63-b035-d6732c19b11c-tls-certs\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.791993 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.791883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-home\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.791993 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.791915    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzprp\" (UniqueName: \"kubernetes.io/projected/30e44d6e-1711-4d63-b035-d6732c19b11c-kube-api-access-kzprp\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.791993 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.791943    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-tmp-dir\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.892892 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.892858    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-home\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.892907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzprp\" (UniqueName: \"kubernetes.io/projected/30e44d6e-1711-4d63-b035-d6732c19b11c-kube-api-access-kzprp\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.892938    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-tmp-dir\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.892993    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-dshm\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893083 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.893024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-model-cache\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893296 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.893104    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e44d6e-1711-4d63-b035-d6732c19b11c-tls-certs\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893296 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.893282    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-home\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893400 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.893323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-tmp-dir\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.893498 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.893469    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-model-cache\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.895452 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.895427    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-dshm\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.895739 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.895722    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e44d6e-1711-4d63-b035-d6732c19b11c-tls-certs\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.901124 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.901104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzprp\" (UniqueName: \"kubernetes.io/projected/30e44d6e-1711-4d63-b035-d6732c19b11c-kube-api-access-kzprp\") pod \"e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") " pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:01.988008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:01.987963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:02.122431 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:02.122409    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"]
Jul 29 19:54:02.124018 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:54:02.123990    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30e44d6e_1711_4d63_b035_d6732c19b11c.slice/crio-63fada7ebeff88e7ed36c9358118150cd1d01fe2676c7c0124ac0f20888dfeb1 WatchSource:0}: Error finding container 63fada7ebeff88e7ed36c9358118150cd1d01fe2676c7c0124ac0f20888dfeb1: Status 404 returned error can't find the container with id 63fada7ebeff88e7ed36c9358118150cd1d01fe2676c7c0124ac0f20888dfeb1
Jul 29 19:54:02.228411 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:02.228377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn" event={"ID":"30e44d6e-1711-4d63-b035-d6732c19b11c","Type":"ContainerStarted","Data":"39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536"}
Jul 29 19:54:02.228612 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:02.228420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn" event={"ID":"30e44d6e-1711-4d63-b035-d6732c19b11c","Type":"ContainerStarted","Data":"63fada7ebeff88e7ed36c9358118150cd1d01fe2676c7c0124ac0f20888dfeb1"}
Jul 29 19:54:02.246061 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:02.245969    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn" podStartSLOduration=1.245952896 podStartE2EDuration="1.245952896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:54:02.243802985 +0000 UTC m=+1448.202297880" watchObservedRunningTime="2026-07-29 19:54:02.245952896 +0000 UTC m=+1448.204447794"
Jul 29 19:54:11.988984 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:11.988948    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:11.989377 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:11.989042    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:12.002197 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:12.002167    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:12.286124 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:12.286033    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:54:49.499061 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:49.499025    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"]
Jul 29 19:54:54.595043 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:54.595011    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:54:54.595757 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:54:54.595735    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:55:04.519525 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.519467    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn" podUID="30e44d6e-1711-4d63-b035-d6732c19b11c" containerName="main" containerID="cri-o://39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536" gracePeriod=45
Jul 29 19:55:04.771310 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.771245    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:55:04.856855 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.856820    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-home\") pod \"30e44d6e-1711-4d63-b035-d6732c19b11c\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") "
Jul 29 19:55:04.857023 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.856884    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e44d6e-1711-4d63-b035-d6732c19b11c-tls-certs\") pod \"30e44d6e-1711-4d63-b035-d6732c19b11c\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") "
Jul 29 19:55:04.857023 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.856934    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-tmp-dir\") pod \"30e44d6e-1711-4d63-b035-d6732c19b11c\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") "
Jul 29 19:55:04.857023 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.856968    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-model-cache\") pod \"30e44d6e-1711-4d63-b035-d6732c19b11c\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") "
Jul 29 19:55:04.857197 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857048    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-dshm\") pod \"30e44d6e-1711-4d63-b035-d6732c19b11c\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") "
Jul 29 19:55:04.857197 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857063    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-home" (OuterVolumeSpecName: "home") pod "30e44d6e-1711-4d63-b035-d6732c19b11c" (UID: "30e44d6e-1711-4d63-b035-d6732c19b11c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:55:04.857197 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857074    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzprp\" (UniqueName: \"kubernetes.io/projected/30e44d6e-1711-4d63-b035-d6732c19b11c-kube-api-access-kzprp\") pod \"30e44d6e-1711-4d63-b035-d6732c19b11c\" (UID: \"30e44d6e-1711-4d63-b035-d6732c19b11c\") "
Jul 29 19:55:04.857343 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857280    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30e44d6e-1711-4d63-b035-d6732c19b11c" (UID: "30e44d6e-1711-4d63-b035-d6732c19b11c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:55:04.857343 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857317    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-model-cache" (OuterVolumeSpecName: "model-cache") pod "30e44d6e-1711-4d63-b035-d6732c19b11c" (UID: "30e44d6e-1711-4d63-b035-d6732c19b11c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:55:04.857471 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857453    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:55:04.857520 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857477    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:55:04.857520 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.857493    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:55:04.859419 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.859389    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-dshm" (OuterVolumeSpecName: "dshm") pod "30e44d6e-1711-4d63-b035-d6732c19b11c" (UID: "30e44d6e-1711-4d63-b035-d6732c19b11c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 19:55:04.859693 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.859668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e44d6e-1711-4d63-b035-d6732c19b11c-kube-api-access-kzprp" (OuterVolumeSpecName: "kube-api-access-kzprp") pod "30e44d6e-1711-4d63-b035-d6732c19b11c" (UID: "30e44d6e-1711-4d63-b035-d6732c19b11c"). InnerVolumeSpecName "kube-api-access-kzprp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:55:04.859771 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.859742    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e44d6e-1711-4d63-b035-d6732c19b11c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30e44d6e-1711-4d63-b035-d6732c19b11c" (UID: "30e44d6e-1711-4d63-b035-d6732c19b11c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:55:04.958286 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.958254    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e44d6e-1711-4d63-b035-d6732c19b11c-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:55:04.958286 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.958281    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e44d6e-1711-4d63-b035-d6732c19b11c-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:55:04.958286 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:04.958289    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzprp\" (UniqueName: \"kubernetes.io/projected/30e44d6e-1711-4d63-b035-d6732c19b11c-kube-api-access-kzprp\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:55:05.524091 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.524055    2555 generic.go:356] "Generic (PLEG): container finished" podID="30e44d6e-1711-4d63-b035-d6732c19b11c" containerID="39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536" exitCode=0
Jul 29 19:55:05.524608 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.524099    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"
Jul 29 19:55:05.524608 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.524149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn" event={"ID":"30e44d6e-1711-4d63-b035-d6732c19b11c","Type":"ContainerDied","Data":"39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536"}
Jul 29 19:55:05.524608 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.524203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn" event={"ID":"30e44d6e-1711-4d63-b035-d6732c19b11c","Type":"ContainerDied","Data":"63fada7ebeff88e7ed36c9358118150cd1d01fe2676c7c0124ac0f20888dfeb1"}
Jul 29 19:55:05.524608 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.524231    2555 scope.go:117] "RemoveContainer" containerID="39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536"
Jul 29 19:55:05.533637 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.533618    2555 scope.go:117] "RemoveContainer" containerID="39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536"
Jul 29 19:55:05.533898 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:55:05.533877    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536\": container with ID starting with 39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536 not found: ID does not exist" containerID="39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536"
Jul 29 19:55:05.533973 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.533904    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536"} err="failed to get container status \"39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536\": rpc error: code = NotFound desc = could not find container \"39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536\": container with ID starting with 39666065c4989d7ba6d0184a1f4f9065abc1cbae4f65a87947ee049fef920536 not found: ID does not exist"
Jul 29 19:55:05.544824 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.544797    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"]
Jul 29 19:55:05.557432 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:05.557367    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fa67c245-kserve-7cdfd77cd7-m8kpn"]
Jul 29 19:55:06.658533 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:55:06.658496    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e44d6e-1711-4d63-b035-d6732c19b11c" path="/var/lib/kubelet/pods/30e44d6e-1711-4d63-b035-d6732c19b11c/volumes"
Jul 29 19:58:27.197405 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.197308    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"]
Jul 29 19:58:27.197932 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.197914    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e44d6e-1711-4d63-b035-d6732c19b11c" containerName="main"
Jul 29 19:58:27.197981 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.197935    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e44d6e-1711-4d63-b035-d6732c19b11c" containerName="main"
Jul 29 19:58:27.198028 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.198013    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e44d6e-1711-4d63-b035-d6732c19b11c" containerName="main"
Jul 29 19:58:27.200985 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.200964    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.202847 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.202827    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c9fa71cf-serving-cert\""
Jul 29 19:58:27.202948 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.202882    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 19:58:27.210264 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.209348    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"]
Jul 29 19:58:27.298134 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.298102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s664\" (UniqueName: \"kubernetes.io/projected/0a71b77c-f3fc-467c-8754-7e493b44ba31-kube-api-access-9s664\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.298295 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.298150    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.298295 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.298211    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71b77c-f3fc-467c-8754-7e493b44ba31-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.399222 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.399186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s664\" (UniqueName: \"kubernetes.io/projected/0a71b77c-f3fc-467c-8754-7e493b44ba31-kube-api-access-9s664\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.399378 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.399232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.399378 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.399261    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71b77c-f3fc-467c-8754-7e493b44ba31-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.399378 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:58:27.399361    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-serving-cert: secret "maas-api-e2e-ait-c9fa71cf-serving-cert" not found
Jul 29 19:58:27.399485 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:58:27.399442    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls podName:0a71b77c-f3fc-467c-8754-7e493b44ba31 nodeName:}" failed. No retries permitted until 2026-07-29 19:58:27.899422655 +0000 UTC m=+1713.857917528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls") pod "maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" (UID: "0a71b77c-f3fc-467c-8754-7e493b44ba31") : secret "maas-api-e2e-ait-c9fa71cf-serving-cert" not found
Jul 29 19:58:27.399989 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.399971    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71b77c-f3fc-467c-8754-7e493b44ba31-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.409932 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.409908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s664\" (UniqueName: \"kubernetes.io/projected/0a71b77c-f3fc-467c-8754-7e493b44ba31-kube-api-access-9s664\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.884192 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.884160    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"]
Jul 29 19:58:27.887857 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.887842    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:27.889870 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.889853    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c9fa71cf\""
Jul 29 19:58:27.899102 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.899072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"]
Jul 29 19:58:27.905284 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.905251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:27.907843 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:27.907822    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:28.006471 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.006428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscg5\" (UniqueName: \"kubernetes.io/projected/2818f99b-291e-490d-a889-7f980ec2591e-kube-api-access-sscg5\") pod \"payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.006471 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.006474    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2818f99b-291e-490d-a889-7f980ec2591e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.107528 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.107489    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscg5\" (UniqueName: \"kubernetes.io/projected/2818f99b-291e-490d-a889-7f980ec2591e-kube-api-access-sscg5\") pod \"payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.107528 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.107531    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2818f99b-291e-490d-a889-7f980ec2591e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.108188 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.108167    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2818f99b-291e-490d-a889-7f980ec2591e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.113911 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.113893    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:28.116618 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.116558    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscg5\" (UniqueName: \"kubernetes.io/projected/2818f99b-291e-490d-a889-7f980ec2591e-kube-api-access-sscg5\") pod \"payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.200624 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.200567    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:28.267037 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.266829    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"]
Jul 29 19:58:28.271271 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:58:28.271231    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a71b77c_f3fc_467c_8754_7e493b44ba31.slice/crio-bf4731d2ccf47579cb73934766267756948e7b774baad44b518df10cb85b591c WatchSource:0}: Error finding container bf4731d2ccf47579cb73934766267756948e7b774baad44b518df10cb85b591c: Status 404 returned error can't find the container with id bf4731d2ccf47579cb73934766267756948e7b774baad44b518df10cb85b591c
Jul 29 19:58:28.272374 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.272353    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 19:58:28.475090 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.475051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" event={"ID":"0a71b77c-f3fc-467c-8754-7e493b44ba31","Type":"ContainerStarted","Data":"bf4731d2ccf47579cb73934766267756948e7b774baad44b518df10cb85b591c"}
Jul 29 19:58:28.557472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:28.557433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"]
Jul 29 19:58:28.559561 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:58:28.559522    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2818f99b_291e_490d_a889_7f980ec2591e.slice/crio-cfd90c76d4ba4a8e80e1efee6deedd21c10e02a01629975b0bfe33333797ed82 WatchSource:0}: Error finding container cfd90c76d4ba4a8e80e1efee6deedd21c10e02a01629975b0bfe33333797ed82: Status 404 returned error can't find the container with id cfd90c76d4ba4a8e80e1efee6deedd21c10e02a01629975b0bfe33333797ed82
Jul 29 19:58:29.481297 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.481257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" event={"ID":"0a71b77c-f3fc-467c-8754-7e493b44ba31","Type":"ContainerStarted","Data":"afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745"}
Jul 29 19:58:29.481863 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.481363    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:29.482908 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.482873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk" event={"ID":"2818f99b-291e-490d-a889-7f980ec2591e","Type":"ContainerStarted","Data":"bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d"}
Jul 29 19:58:29.483015 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.482921    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk" event={"ID":"2818f99b-291e-490d-a889-7f980ec2591e","Type":"ContainerStarted","Data":"cfd90c76d4ba4a8e80e1efee6deedd21c10e02a01629975b0bfe33333797ed82"}
Jul 29 19:58:29.483015 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.482965    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:29.502369 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.502326    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" podStartSLOduration=1.875490348 podStartE2EDuration="2.502310418s" totalImagesPullingTime="626.82007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:58:29.500491244 +0000 UTC m=+1715.458986163" watchObservedRunningTime="2026-07-29 19:58:29.502310418 +0000 UTC m=+1715.460805313"
Jul 29 19:58:29.516823 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:29.516773    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk" podStartSLOduration=2.516758896 podStartE2EDuration="2.516758896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:58:29.515093783 +0000 UTC m=+1715.473588699" watchObservedRunningTime="2026-07-29 19:58:29.516758896 +0000 UTC m=+1715.475253789"
Jul 29 19:58:35.492905 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:35.492872    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:40.490231 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:40.490196    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:57.881689 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:57.881655    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"]
Jul 29 19:58:57.882095 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:57.881859    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" podUID="0a71b77c-f3fc-467c-8754-7e493b44ba31" containerName="maas-api" containerID="cri-o://afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745" gracePeriod=30
Jul 29 19:58:57.941967 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:57.941928    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"]
Jul 29 19:58:57.942190 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:57.942164    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk" podUID="2818f99b-291e-490d-a889-7f980ec2591e" containerName="payload-pre-processing" containerID="cri-o://bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d" gracePeriod=30
Jul 29 19:58:58.195716 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.195692    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:58.219145 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.219080    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:58.283525 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.283493    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sscg5\" (UniqueName: \"kubernetes.io/projected/2818f99b-291e-490d-a889-7f980ec2591e-kube-api-access-sscg5\") pod \"2818f99b-291e-490d-a889-7f980ec2591e\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") "
Jul 29 19:58:58.283795 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.283551    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2818f99b-291e-490d-a889-7f980ec2591e-plugins-config-volume\") pod \"2818f99b-291e-490d-a889-7f980ec2591e\" (UID: \"2818f99b-291e-490d-a889-7f980ec2591e\") "
Jul 29 19:58:58.283795 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.283650    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s664\" (UniqueName: \"kubernetes.io/projected/0a71b77c-f3fc-467c-8754-7e493b44ba31-kube-api-access-9s664\") pod \"0a71b77c-f3fc-467c-8754-7e493b44ba31\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") "
Jul 29 19:58:58.283795 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.283676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls\") pod \"0a71b77c-f3fc-467c-8754-7e493b44ba31\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") "
Jul 29 19:58:58.283795 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.283708    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71b77c-f3fc-467c-8754-7e493b44ba31-openshift-service-ca\") pod \"0a71b77c-f3fc-467c-8754-7e493b44ba31\" (UID: \"0a71b77c-f3fc-467c-8754-7e493b44ba31\") "
Jul 29 19:58:58.284022 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.283942    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2818f99b-291e-490d-a889-7f980ec2591e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2818f99b-291e-490d-a889-7f980ec2591e" (UID: "2818f99b-291e-490d-a889-7f980ec2591e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:58.284151 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.284126    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a71b77c-f3fc-467c-8754-7e493b44ba31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a71b77c-f3fc-467c-8754-7e493b44ba31" (UID: "0a71b77c-f3fc-467c-8754-7e493b44ba31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:58:58.285986 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.285964    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a71b77c-f3fc-467c-8754-7e493b44ba31" (UID: "0a71b77c-f3fc-467c-8754-7e493b44ba31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:58:58.285986 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.285967    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a71b77c-f3fc-467c-8754-7e493b44ba31-kube-api-access-9s664" (OuterVolumeSpecName: "kube-api-access-9s664") pod "0a71b77c-f3fc-467c-8754-7e493b44ba31" (UID: "0a71b77c-f3fc-467c-8754-7e493b44ba31"). InnerVolumeSpecName "kube-api-access-9s664". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:58.286091 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.286005    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2818f99b-291e-490d-a889-7f980ec2591e-kube-api-access-sscg5" (OuterVolumeSpecName: "kube-api-access-sscg5") pod "2818f99b-291e-490d-a889-7f980ec2591e" (UID: "2818f99b-291e-490d-a889-7f980ec2591e"). InnerVolumeSpecName "kube-api-access-sscg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:58:58.384892 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.384857    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscg5\" (UniqueName: \"kubernetes.io/projected/2818f99b-291e-490d-a889-7f980ec2591e-kube-api-access-sscg5\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.384892 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.384884    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2818f99b-291e-490d-a889-7f980ec2591e-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.384892 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.384894    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s664\" (UniqueName: \"kubernetes.io/projected/0a71b77c-f3fc-467c-8754-7e493b44ba31-kube-api-access-9s664\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.385106 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.384903    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a71b77c-f3fc-467c-8754-7e493b44ba31-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.385106 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.384911    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a71b77c-f3fc-467c-8754-7e493b44ba31-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:58:58.618728 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.618698    2555 generic.go:356] "Generic (PLEG): container finished" podID="0a71b77c-f3fc-467c-8754-7e493b44ba31" containerID="afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745" exitCode=0
Jul 29 19:58:58.618931 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.618766    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"
Jul 29 19:58:58.618931 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.618790    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" event={"ID":"0a71b77c-f3fc-467c-8754-7e493b44ba31","Type":"ContainerDied","Data":"afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745"}
Jul 29 19:58:58.618931 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.618855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh" event={"ID":"0a71b77c-f3fc-467c-8754-7e493b44ba31","Type":"ContainerDied","Data":"bf4731d2ccf47579cb73934766267756948e7b774baad44b518df10cb85b591c"}
Jul 29 19:58:58.618931 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.618888    2555 scope.go:117] "RemoveContainer" containerID="afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745"
Jul 29 19:58:58.620137 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.620108    2555 generic.go:356] "Generic (PLEG): container finished" podID="2818f99b-291e-490d-a889-7f980ec2591e" containerID="bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d" exitCode=0
Jul 29 19:58:58.620247 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.620164    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"
Jul 29 19:58:58.620247 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.620194    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk" event={"ID":"2818f99b-291e-490d-a889-7f980ec2591e","Type":"ContainerDied","Data":"bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d"}
Jul 29 19:58:58.620247 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.620244    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk" event={"ID":"2818f99b-291e-490d-a889-7f980ec2591e","Type":"ContainerDied","Data":"cfd90c76d4ba4a8e80e1efee6deedd21c10e02a01629975b0bfe33333797ed82"}
Jul 29 19:58:58.628902 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.628881    2555 scope.go:117] "RemoveContainer" containerID="afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745"
Jul 29 19:58:58.629167 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:58:58.629147    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745\": container with ID starting with afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745 not found: ID does not exist" containerID="afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745"
Jul 29 19:58:58.629240 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.629174    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745"} err="failed to get container status \"afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745\": rpc error: code = NotFound desc = could not find container \"afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745\": container with ID starting with afb3594874c78f265cf61e935f5cd6a1f3ec729166a121757c7fc5a154a07745 not found: ID does not exist"
Jul 29 19:58:58.629240 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.629190    2555 scope.go:117] "RemoveContainer" containerID="bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d"
Jul 29 19:58:58.637175 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.637159    2555 scope.go:117] "RemoveContainer" containerID="bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d"
Jul 29 19:58:58.637413 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:58:58.637396    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d\": container with ID starting with bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d not found: ID does not exist" containerID="bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d"
Jul 29 19:58:58.637469 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.637420    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d"} err="failed to get container status \"bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d\": rpc error: code = NotFound desc = could not find container \"bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d\": container with ID starting with bc46fadfb7502cdcf103a0738fb0e4ea90c5fba54dd192f4bfcb28e5b2c7f05d not found: ID does not exist"
Jul 29 19:58:58.646608 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.646567    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"]
Jul 29 19:58:58.651472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.651448    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa71cf-54c6cb9fcb-7wbnh"]
Jul 29 19:58:58.665887 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.665819    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a71b77c-f3fc-467c-8754-7e493b44ba31" path="/var/lib/kubelet/pods/0a71b77c-f3fc-467c-8754-7e493b44ba31/volumes"
Jul 29 19:58:58.666380 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.666359    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"]
Jul 29 19:58:58.672271 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:58:58.672246    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9fa71cf-59b9b6866d-7n6kk"]
Jul 29 19:59:00.658552 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:00.658512    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2818f99b-291e-490d-a889-7f980ec2591e" path="/var/lib/kubelet/pods/2818f99b-291e-490d-a889-7f980ec2591e/volumes"
Jul 29 19:59:01.466197 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466160    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"]
Jul 29 19:59:01.466593 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466562    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a71b77c-f3fc-467c-8754-7e493b44ba31" containerName="maas-api"
Jul 29 19:59:01.466687 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466595    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a71b77c-f3fc-467c-8754-7e493b44ba31" containerName="maas-api"
Jul 29 19:59:01.466687 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466613    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2818f99b-291e-490d-a889-7f980ec2591e" containerName="payload-pre-processing"
Jul 29 19:59:01.466687 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466621    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2818f99b-291e-490d-a889-7f980ec2591e" containerName="payload-pre-processing"
Jul 29 19:59:01.466845 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466696    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a71b77c-f3fc-467c-8754-7e493b44ba31" containerName="maas-api"
Jul 29 19:59:01.466845 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.466710    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2818f99b-291e-490d-a889-7f980ec2591e" containerName="payload-pre-processing"
Jul 29 19:59:01.471472 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.471447    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.473488 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.473466    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 19:59:01.473644 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.473498    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fbe86793-serving-cert\""
Jul 29 19:59:01.479291 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.479254    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"]
Jul 29 19:59:01.511749 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.511720    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a05772-e3d6-4f36-a673-903fd4bf42a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.511889 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.511765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8ds\" (UniqueName: \"kubernetes.io/projected/74a05772-e3d6-4f36-a673-903fd4bf42a3-kube-api-access-6n8ds\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.511889 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.511840    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.612922 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.612888    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a05772-e3d6-4f36-a673-903fd4bf42a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.613084 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.612934    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8ds\" (UniqueName: \"kubernetes.io/projected/74a05772-e3d6-4f36-a673-903fd4bf42a3-kube-api-access-6n8ds\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.613084 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.612980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.613161 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:01.613126    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-serving-cert: secret "maas-api-e2e-migrate-fbe86793-serving-cert" not found
Jul 29 19:59:01.613204 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:01.613194    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls podName:74a05772-e3d6-4f36-a673-903fd4bf42a3 nodeName:}" failed. No retries permitted until 2026-07-29 19:59:02.113174332 +0000 UTC m=+1748.071669224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls") pod "maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" (UID: "74a05772-e3d6-4f36-a673-903fd4bf42a3") : secret "maas-api-e2e-migrate-fbe86793-serving-cert" not found
Jul 29 19:59:01.613566 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.613545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a05772-e3d6-4f36-a673-903fd4bf42a3-openshift-service-ca\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:01.622800 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:01.622776    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8ds\" (UniqueName: \"kubernetes.io/projected/74a05772-e3d6-4f36-a673-903fd4bf42a3-kube-api-access-6n8ds\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:02.118131 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.118097    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:02.120714 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.120686    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls\") pod \"maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:02.256985 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.256948    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"]
Jul 29 19:59:02.260670 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.260650    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.262602 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.262570    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fbe86793-dockercfg-k2xkg\""
Jul 29 19:59:02.262695 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.262672    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fbe86793\""
Jul 29 19:59:02.275110 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.275089    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"]
Jul 29 19:59:02.318420 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.318381    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"]
Jul 29 19:59:02.320070 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.320040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vcc\" (UniqueName: \"kubernetes.io/projected/fc4ef173-019d-45fc-93fe-39c73a279822-kube-api-access-b9vcc\") pod \"payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.320221 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.320095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef173-019d-45fc-93fe-39c73a279822-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.321841 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.321822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.333762 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.333732    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"]
Jul 29 19:59:02.385537 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.385453    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:02.421247 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.421213    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vcc\" (UniqueName: \"kubernetes.io/projected/fc4ef173-019d-45fc-93fe-39c73a279822-kube-api-access-b9vcc\") pod \"payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.421430 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.421262    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef173-019d-45fc-93fe-39c73a279822-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.421430 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.421313    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6rg\" (UniqueName: \"kubernetes.io/projected/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-kube-api-access-2w6rg\") pod \"payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.421430 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.421392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.422380 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.422347    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef173-019d-45fc-93fe-39c73a279822-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.430772 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.430717    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vcc\" (UniqueName: \"kubernetes.io/projected/fc4ef173-019d-45fc-93fe-39c73a279822-kube-api-access-b9vcc\") pod \"payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.521910 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.521873    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.522055 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.521986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6rg\" (UniqueName: \"kubernetes.io/projected/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-kube-api-access-2w6rg\") pod \"payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.522663 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.522643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.530947 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.530924    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6rg\" (UniqueName: \"kubernetes.io/projected/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-kube-api-access-2w6rg\") pod \"payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.532872 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.532846    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"]
Jul 29 19:59:02.536403 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:59:02.536373    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a05772_e3d6_4f36_a673_903fd4bf42a3.slice/crio-5bf42de0a1bc665585cbb50f9890af78e3b831a0fcf3138c24b6e2e8640d24fa WatchSource:0}: Error finding container 5bf42de0a1bc665585cbb50f9890af78e3b831a0fcf3138c24b6e2e8640d24fa: Status 404 returned error can't find the container with id 5bf42de0a1bc665585cbb50f9890af78e3b831a0fcf3138c24b6e2e8640d24fa
Jul 29 19:59:02.571269 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.571236    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:02.634677 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.634644    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:02.662378 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.662340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" event={"ID":"74a05772-e3d6-4f36-a673-903fd4bf42a3","Type":"ContainerStarted","Data":"5bf42de0a1bc665585cbb50f9890af78e3b831a0fcf3138c24b6e2e8640d24fa"}
Jul 29 19:59:02.726318 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.726176    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"]
Jul 29 19:59:02.729247 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:59:02.729214    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4ef173_019d_45fc_93fe_39c73a279822.slice/crio-ef3612ca8938f9e67bdba2175ae91a2751f87f843671a9ac93736782011b0e8b WatchSource:0}: Error finding container ef3612ca8938f9e67bdba2175ae91a2751f87f843671a9ac93736782011b0e8b: Status 404 returned error can't find the container with id ef3612ca8938f9e67bdba2175ae91a2751f87f843671a9ac93736782011b0e8b
Jul 29 19:59:02.797453 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:02.797427    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"]
Jul 29 19:59:02.799755 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:59:02.799715    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899ad56c_bb1c_47b7_9d7d_72c93bcc8fa9.slice/crio-28ceaa61418716b60e1af945628fd97d68b61767a2ebedceb29bc50479f46724 WatchSource:0}: Error finding container 28ceaa61418716b60e1af945628fd97d68b61767a2ebedceb29bc50479f46724: Status 404 returned error can't find the container with id 28ceaa61418716b60e1af945628fd97d68b61767a2ebedceb29bc50479f46724
Jul 29 19:59:03.662961 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.662923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" event={"ID":"74a05772-e3d6-4f36-a673-903fd4bf42a3","Type":"ContainerStarted","Data":"2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62"}
Jul 29 19:59:03.663485 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.662997    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:03.664708 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.664677    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc" event={"ID":"fc4ef173-019d-45fc-93fe-39c73a279822","Type":"ContainerStarted","Data":"c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a"}
Jul 29 19:59:03.664840 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.664730    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc" event={"ID":"fc4ef173-019d-45fc-93fe-39c73a279822","Type":"ContainerStarted","Data":"ef3612ca8938f9e67bdba2175ae91a2751f87f843671a9ac93736782011b0e8b"}
Jul 29 19:59:03.664840 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.664758    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:03.666307 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.666284    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh" event={"ID":"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9","Type":"ContainerStarted","Data":"5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647"}
Jul 29 19:59:03.666394 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.666323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh" event={"ID":"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9","Type":"ContainerStarted","Data":"28ceaa61418716b60e1af945628fd97d68b61767a2ebedceb29bc50479f46724"}
Jul 29 19:59:03.666394 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.666365    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:03.681070 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.680994    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" podStartSLOduration=2.275126403 podStartE2EDuration="2.680975363s" totalImagesPullingTime="405.84896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:03.67821116 +0000 UTC m=+1749.636706080" watchObservedRunningTime="2026-07-29 19:59:03.680975363 +0000 UTC m=+1749.639470261"
Jul 29 19:59:03.695029 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.694986    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc" podStartSLOduration=1.6949740009999998 podStartE2EDuration="1.694974001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:03.693175079 +0000 UTC m=+1749.651669998" watchObservedRunningTime="2026-07-29 19:59:03.694974001 +0000 UTC m=+1749.653468895"
Jul 29 19:59:03.709392 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:03.709352    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh" podStartSLOduration=1.709337879 podStartE2EDuration="1.709337879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:03.707812771 +0000 UTC m=+1749.666307668" watchObservedRunningTime="2026-07-29 19:59:03.709337879 +0000 UTC m=+1749.667832772"
Jul 29 19:59:09.676301 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:09.676271    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:14.673088 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:14.673059    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:14.673513 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:14.673134    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:29.941928 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:29.941894    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"]
Jul 29 19:59:29.942416 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:29.942104    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" podUID="74a05772-e3d6-4f36-a673-903fd4bf42a3" containerName="maas-api" containerID="cri-o://2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62" gracePeriod=30
Jul 29 19:59:30.000194 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.000160    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"]
Jul 29 19:59:30.000421 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.000397    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc" podUID="fc4ef173-019d-45fc-93fe-39c73a279822" containerName="payload-processing" containerID="cri-o://c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a" gracePeriod=30
Jul 29 19:59:30.006115 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.006079    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"]
Jul 29 19:59:30.006390 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.006326    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh" podUID="899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" containerName="payload-pre-processing" containerID="cri-o://5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647" gracePeriod=30
Jul 29 19:59:30.319484 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.319202    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:30.328701 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.328394    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:30.333572 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.333549    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:30.487283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n8ds\" (UniqueName: \"kubernetes.io/projected/74a05772-e3d6-4f36-a673-903fd4bf42a3-kube-api-access-6n8ds\") pod \"74a05772-e3d6-4f36-a673-903fd4bf42a3\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") "
Jul 29 19:59:30.487283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487235    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef173-019d-45fc-93fe-39c73a279822-plugins-config-volume\") pod \"fc4ef173-019d-45fc-93fe-39c73a279822\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") "
Jul 29 19:59:30.487283 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487271    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w6rg\" (UniqueName: \"kubernetes.io/projected/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-kube-api-access-2w6rg\") pod \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") "
Jul 29 19:59:30.487546 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487396    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a05772-e3d6-4f36-a673-903fd4bf42a3-openshift-service-ca\") pod \"74a05772-e3d6-4f36-a673-903fd4bf42a3\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") "
Jul 29 19:59:30.487546 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487460    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls\") pod \"74a05772-e3d6-4f36-a673-903fd4bf42a3\" (UID: \"74a05772-e3d6-4f36-a673-903fd4bf42a3\") "
Jul 29 19:59:30.487546 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487489    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-plugins-config-volume\") pod \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\" (UID: \"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9\") "
Jul 29 19:59:30.487546 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487528    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4ef173-019d-45fc-93fe-39c73a279822-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc4ef173-019d-45fc-93fe-39c73a279822" (UID: "fc4ef173-019d-45fc-93fe-39c73a279822"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:59:30.487789 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487676    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9vcc\" (UniqueName: \"kubernetes.io/projected/fc4ef173-019d-45fc-93fe-39c73a279822-kube-api-access-b9vcc\") pod \"fc4ef173-019d-45fc-93fe-39c73a279822\" (UID: \"fc4ef173-019d-45fc-93fe-39c73a279822\") "
Jul 29 19:59:30.487897 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.487872    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a05772-e3d6-4f36-a673-903fd4bf42a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74a05772-e3d6-4f36-a673-903fd4bf42a3" (UID: "74a05772-e3d6-4f36-a673-903fd4bf42a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:59:30.488178 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.488128    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4ef173-019d-45fc-93fe-39c73a279822-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.488178 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.488156    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a05772-e3d6-4f36-a673-903fd4bf42a3-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.488178 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.488154    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" (UID: "899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 19:59:30.489851 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.489824    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a05772-e3d6-4f36-a673-903fd4bf42a3-kube-api-access-6n8ds" (OuterVolumeSpecName: "kube-api-access-6n8ds") pod "74a05772-e3d6-4f36-a673-903fd4bf42a3" (UID: "74a05772-e3d6-4f36-a673-903fd4bf42a3"). InnerVolumeSpecName "kube-api-access-6n8ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:59:30.489933 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.489897    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4ef173-019d-45fc-93fe-39c73a279822-kube-api-access-b9vcc" (OuterVolumeSpecName: "kube-api-access-b9vcc") pod "fc4ef173-019d-45fc-93fe-39c73a279822" (UID: "fc4ef173-019d-45fc-93fe-39c73a279822"). InnerVolumeSpecName "kube-api-access-b9vcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:59:30.490306 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.490278    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-kube-api-access-2w6rg" (OuterVolumeSpecName: "kube-api-access-2w6rg") pod "899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" (UID: "899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9"). InnerVolumeSpecName "kube-api-access-2w6rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 19:59:30.490399 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.490278    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74a05772-e3d6-4f36-a673-903fd4bf42a3" (UID: "74a05772-e3d6-4f36-a673-903fd4bf42a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 19:59:30.589376 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.589340    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vcc\" (UniqueName: \"kubernetes.io/projected/fc4ef173-019d-45fc-93fe-39c73a279822-kube-api-access-b9vcc\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.589376 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.589372    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8ds\" (UniqueName: \"kubernetes.io/projected/74a05772-e3d6-4f36-a673-903fd4bf42a3-kube-api-access-6n8ds\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.589376 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.589385    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6rg\" (UniqueName: \"kubernetes.io/projected/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-kube-api-access-2w6rg\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.589645 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.589398    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74a05772-e3d6-4f36-a673-903fd4bf42a3-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.589645 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.589409    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 19:59:30.798008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.797912    2555 generic.go:356] "Generic (PLEG): container finished" podID="74a05772-e3d6-4f36-a673-903fd4bf42a3" containerID="2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62" exitCode=0
Jul 29 19:59:30.798008 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.797967    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"
Jul 29 19:59:30.798233 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.798004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" event={"ID":"74a05772-e3d6-4f36-a673-903fd4bf42a3","Type":"ContainerDied","Data":"2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62"}
Jul 29 19:59:30.798233 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.798053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s" event={"ID":"74a05772-e3d6-4f36-a673-903fd4bf42a3","Type":"ContainerDied","Data":"5bf42de0a1bc665585cbb50f9890af78e3b831a0fcf3138c24b6e2e8640d24fa"}
Jul 29 19:59:30.798233 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.798083    2555 scope.go:117] "RemoveContainer" containerID="2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62"
Jul 29 19:59:30.799373 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.799353    2555 generic.go:356] "Generic (PLEG): container finished" podID="fc4ef173-019d-45fc-93fe-39c73a279822" containerID="c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a" exitCode=0
Jul 29 19:59:30.799459 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.799444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc" event={"ID":"fc4ef173-019d-45fc-93fe-39c73a279822","Type":"ContainerDied","Data":"c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a"}
Jul 29 19:59:30.799506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.799488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc" event={"ID":"fc4ef173-019d-45fc-93fe-39c73a279822","Type":"ContainerDied","Data":"ef3612ca8938f9e67bdba2175ae91a2751f87f843671a9ac93736782011b0e8b"}
Jul 29 19:59:30.799506 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.799446    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"
Jul 29 19:59:30.800899 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.800876    2555 generic.go:356] "Generic (PLEG): container finished" podID="899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" containerID="5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647" exitCode=0
Jul 29 19:59:30.801004 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.800935    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh" event={"ID":"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9","Type":"ContainerDied","Data":"5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647"}
Jul 29 19:59:30.801004 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.800959    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"
Jul 29 19:59:30.801004 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.800969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh" event={"ID":"899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9","Type":"ContainerDied","Data":"28ceaa61418716b60e1af945628fd97d68b61767a2ebedceb29bc50479f46724"}
Jul 29 19:59:30.810686 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.810667    2555 scope.go:117] "RemoveContainer" containerID="2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62"
Jul 29 19:59:30.811008 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:30.810989    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62\": container with ID starting with 2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62 not found: ID does not exist" containerID="2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62"
Jul 29 19:59:30.811091 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.811023    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62"} err="failed to get container status \"2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62\": rpc error: code = NotFound desc = could not find container \"2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62\": container with ID starting with 2e9ef49af2fdc5cf90a177be7a886ce4b68d177193e6772cde10b84f5ec9da62 not found: ID does not exist"
Jul 29 19:59:30.811091 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.811049    2555 scope.go:117] "RemoveContainer" containerID="c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a"
Jul 29 19:59:30.818733 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.818695    2555 scope.go:117] "RemoveContainer" containerID="c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a"
Jul 29 19:59:30.818997 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:30.818973    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a\": container with ID starting with c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a not found: ID does not exist" containerID="c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a"
Jul 29 19:59:30.819052 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.819006    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a"} err="failed to get container status \"c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a\": rpc error: code = NotFound desc = could not find container \"c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a\": container with ID starting with c8ac287cb4a5497991cff076847f99de985936c225ee9d7cc61771cb4ff31e5a not found: ID does not exist"
Jul 29 19:59:30.819052 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.819021    2555 scope.go:117] "RemoveContainer" containerID="5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647"
Jul 29 19:59:30.824362 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.824339    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"]
Jul 29 19:59:30.827058 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.826821    2555 scope.go:117] "RemoveContainer" containerID="5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647"
Jul 29 19:59:30.827204 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:30.827184    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647\": container with ID starting with 5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647 not found: ID does not exist" containerID="5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647"
Jul 29 19:59:30.827270 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.827215    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647"} err="failed to get container status \"5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647\": rpc error: code = NotFound desc = could not find container \"5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647\": container with ID starting with 5009419f85c768f40097efd3bc618421618520a8716f848fbac232b600c09647 not found: ID does not exist"
Jul 29 19:59:30.828508 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.828490    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbe86793-78c8b7dcc7-r484s"]
Jul 29 19:59:30.839210 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.839186    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"]
Jul 29 19:59:30.845521 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.845499    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbe86793-85cbcd7cf7-7kgsc"]
Jul 29 19:59:30.856488 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.856466    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"]
Jul 29 19:59:30.860751 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:30.860734    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fbe86793-5889dcd84-9ggzh"]
Jul 29 19:59:32.659601 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:32.659541    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a05772-e3d6-4f36-a673-903fd4bf42a3" path="/var/lib/kubelet/pods/74a05772-e3d6-4f36-a673-903fd4bf42a3/volumes"
Jul 29 19:59:32.660102 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:32.660084    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" path="/var/lib/kubelet/pods/899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9/volumes"
Jul 29 19:59:32.660533 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:32.660517    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4ef173-019d-45fc-93fe-39c73a279822" path="/var/lib/kubelet/pods/fc4ef173-019d-45fc-93fe-39c73a279822/volumes"
Jul 29 19:59:33.517409 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517368    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"]
Jul 29 19:59:33.517790 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517775    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a05772-e3d6-4f36-a673-903fd4bf42a3" containerName="maas-api"
Jul 29 19:59:33.517839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517792    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a05772-e3d6-4f36-a673-903fd4bf42a3" containerName="maas-api"
Jul 29 19:59:33.517839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517807    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4ef173-019d-45fc-93fe-39c73a279822" containerName="payload-processing"
Jul 29 19:59:33.517839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517813    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4ef173-019d-45fc-93fe-39c73a279822" containerName="payload-processing"
Jul 29 19:59:33.517839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517820    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" containerName="payload-pre-processing"
Jul 29 19:59:33.517839 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517826    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" containerName="payload-pre-processing"
Jul 29 19:59:33.517982 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517887    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a05772-e3d6-4f36-a673-903fd4bf42a3" containerName="maas-api"
Jul 29 19:59:33.517982 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517896    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4ef173-019d-45fc-93fe-39c73a279822" containerName="payload-processing"
Jul 29 19:59:33.517982 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.517902    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="899ad56c-bb1c-47b7-9d7d-72c93bcc8fa9" containerName="payload-pre-processing"
Jul 29 19:59:33.522492 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.522472    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.524422 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.524406    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-73e856f2-serving-cert\""
Jul 29 19:59:33.524502 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.524468    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 19:59:33.529467 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.529441    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"]
Jul 29 19:59:33.619801 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.619765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b8994a-b250-43d2-9b62-339fd6089542-openshift-service-ca\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.619965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.619820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwkw\" (UniqueName: \"kubernetes.io/projected/05b8994a-b250-43d2-9b62-339fd6089542-kube-api-access-8pwkw\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.619965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.619847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.720826 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.720790    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.721276 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.720904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b8994a-b250-43d2-9b62-339fd6089542-openshift-service-ca\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.721276 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:33.720930    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-serving-cert: secret "maas-api-e2e-ait-73e856f2-serving-cert" not found
Jul 29 19:59:33.721276 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.720972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwkw\" (UniqueName: \"kubernetes.io/projected/05b8994a-b250-43d2-9b62-339fd6089542-kube-api-access-8pwkw\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.721276 ip-10-0-138-139 kubenswrapper[2555]: E0729 19:59:33.720998    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls podName:05b8994a-b250-43d2-9b62-339fd6089542 nodeName:}" failed. No retries permitted until 2026-07-29 19:59:34.220975776 +0000 UTC m=+1780.179470648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls") pod "maas-api-e2e-ait-73e856f2-7bb8775868-98852" (UID: "05b8994a-b250-43d2-9b62-339fd6089542") : secret "maas-api-e2e-ait-73e856f2-serving-cert" not found
Jul 29 19:59:33.721693 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.721670    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b8994a-b250-43d2-9b62-339fd6089542-openshift-service-ca\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:33.730079 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:33.730058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwkw\" (UniqueName: \"kubernetes.io/projected/05b8994a-b250-43d2-9b62-339fd6089542-kube-api-access-8pwkw\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:34.226058 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.226024    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:34.228536 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.228508    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls\") pod \"maas-api-e2e-ait-73e856f2-7bb8775868-98852\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:34.259148 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.259114    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"]
Jul 29 19:59:34.262959 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.262938    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.264965 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.264941    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-73e856f2-dockercfg-652z8\""
Jul 29 19:59:34.265099 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.264976    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-73e856f2\""
Jul 29 19:59:34.273202 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.273176    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"]
Jul 29 19:59:34.326885 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.326849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ec146-213e-4f76-b053-2514577ca7bc-plugins-config-volume\") pod \"payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") " pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.327035 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.326890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkmz\" (UniqueName: \"kubernetes.io/projected/5d9ec146-213e-4f76-b053-2514577ca7bc-kube-api-access-7tkmz\") pod \"payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") " pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.427875 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.427835    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ec146-213e-4f76-b053-2514577ca7bc-plugins-config-volume\") pod \"payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") " pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.428041 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.427885    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkmz\" (UniqueName: \"kubernetes.io/projected/5d9ec146-213e-4f76-b053-2514577ca7bc-kube-api-access-7tkmz\") pod \"payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") " pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.428539 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.428519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ec146-213e-4f76-b053-2514577ca7bc-plugins-config-volume\") pod \"payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") " pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.433785 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.433758    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:34.438279 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.438256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkmz\" (UniqueName: \"kubernetes.io/projected/5d9ec146-213e-4f76-b053-2514577ca7bc-kube-api-access-7tkmz\") pod \"payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") " pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.575148 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.575124    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.575476 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.575451    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"]
Jul 29 19:59:34.577608 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:59:34.577565    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b8994a_b250_43d2_9b62_339fd6089542.slice/crio-17f13d231773a5aac5628bfb628b6f44753af5b7c7d0410350ea6ec8bf124154 WatchSource:0}: Error finding container 17f13d231773a5aac5628bfb628b6f44753af5b7c7d0410350ea6ec8bf124154: Status 404 returned error can't find the container with id 17f13d231773a5aac5628bfb628b6f44753af5b7c7d0410350ea6ec8bf124154
Jul 29 19:59:34.725751 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.725723    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"]
Jul 29 19:59:34.728142 ip-10-0-138-139 kubenswrapper[2555]: W0729 19:59:34.728119    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9ec146_213e_4f76_b053_2514577ca7bc.slice/crio-87df45451f849e72a9d08ba28758887a740634613116b3e69fbd3451391d3cef WatchSource:0}: Error finding container 87df45451f849e72a9d08ba28758887a740634613116b3e69fbd3451391d3cef: Status 404 returned error can't find the container with id 87df45451f849e72a9d08ba28758887a740634613116b3e69fbd3451391d3cef
Jul 29 19:59:34.823992 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.823959    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852" event={"ID":"05b8994a-b250-43d2-9b62-339fd6089542","Type":"ContainerStarted","Data":"17f13d231773a5aac5628bfb628b6f44753af5b7c7d0410350ea6ec8bf124154"}
Jul 29 19:59:34.825826 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.825789    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b" event={"ID":"5d9ec146-213e-4f76-b053-2514577ca7bc","Type":"ContainerStarted","Data":"e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d"}
Jul 29 19:59:34.825984 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.825847    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b" event={"ID":"5d9ec146-213e-4f76-b053-2514577ca7bc","Type":"ContainerStarted","Data":"87df45451f849e72a9d08ba28758887a740634613116b3e69fbd3451391d3cef"}
Jul 29 19:59:34.825984 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.825877    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:34.844140 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:34.844085    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b" podStartSLOduration=0.844065015 podStartE2EDuration="844.065015ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:34.84176208 +0000 UTC m=+1780.800256975" watchObservedRunningTime="2026-07-29 19:59:34.844065015 +0000 UTC m=+1780.802559910"
Jul 29 19:59:35.833006 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:35.832969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852" event={"ID":"05b8994a-b250-43d2-9b62-339fd6089542","Type":"ContainerStarted","Data":"420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7"}
Jul 29 19:59:35.833432 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:35.833188    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:35.851268 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:35.851208    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852" podStartSLOduration=2.455693008 podStartE2EDuration="2.851188657s" totalImagesPullingTime="395.495649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:59:35.849419167 +0000 UTC m=+1781.807914089" watchObservedRunningTime="2026-07-29 19:59:35.851188657 +0000 UTC m=+1781.809683555"
Jul 29 19:59:41.843910 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:41.843878    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 19:59:45.835787 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:45.835700    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 19:59:54.624046 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:54.624018    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 19:59:54.625937 ip-10-0-138-139 kubenswrapper[2555]: I0729 19:59:54.625919    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:00:14.165037 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.164993    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"]
Jul 29 20:00:14.165940 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.165870    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852" podUID="05b8994a-b250-43d2-9b62-339fd6089542" containerName="maas-api" containerID="cri-o://420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7" gracePeriod=30
Jul 29 20:00:14.230123 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.230080    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"]
Jul 29 20:00:14.230403 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.230354    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b" podUID="5d9ec146-213e-4f76-b053-2514577ca7bc" containerName="payload-processing" containerID="cri-o://e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d" gracePeriod=30
Jul 29 20:00:14.484653 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.484628    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 20:00:14.515658 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.515625    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 20:00:14.587022 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.586981    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pwkw\" (UniqueName: \"kubernetes.io/projected/05b8994a-b250-43d2-9b62-339fd6089542-kube-api-access-8pwkw\") pod \"05b8994a-b250-43d2-9b62-339fd6089542\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") "
Jul 29 20:00:14.587214 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.587063    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b8994a-b250-43d2-9b62-339fd6089542-openshift-service-ca\") pod \"05b8994a-b250-43d2-9b62-339fd6089542\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") "
Jul 29 20:00:14.587214 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.587184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls\") pod \"05b8994a-b250-43d2-9b62-339fd6089542\" (UID: \"05b8994a-b250-43d2-9b62-339fd6089542\") "
Jul 29 20:00:14.587491 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.587457    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b8994a-b250-43d2-9b62-339fd6089542-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05b8994a-b250-43d2-9b62-339fd6089542" (UID: "05b8994a-b250-43d2-9b62-339fd6089542"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:00:14.589188 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.589167    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b8994a-b250-43d2-9b62-339fd6089542-kube-api-access-8pwkw" (OuterVolumeSpecName: "kube-api-access-8pwkw") pod "05b8994a-b250-43d2-9b62-339fd6089542" (UID: "05b8994a-b250-43d2-9b62-339fd6089542"). InnerVolumeSpecName "kube-api-access-8pwkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:00:14.589259 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.589243    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05b8994a-b250-43d2-9b62-339fd6089542" (UID: "05b8994a-b250-43d2-9b62-339fd6089542"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:00:14.687998 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.687972    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tkmz\" (UniqueName: \"kubernetes.io/projected/5d9ec146-213e-4f76-b053-2514577ca7bc-kube-api-access-7tkmz\") pod \"5d9ec146-213e-4f76-b053-2514577ca7bc\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") "
Jul 29 20:00:14.688127 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.688070    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ec146-213e-4f76-b053-2514577ca7bc-plugins-config-volume\") pod \"5d9ec146-213e-4f76-b053-2514577ca7bc\" (UID: \"5d9ec146-213e-4f76-b053-2514577ca7bc\") "
Jul 29 20:00:14.688238 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.688227    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05b8994a-b250-43d2-9b62-339fd6089542-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:14.688295 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.688240    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwkw\" (UniqueName: \"kubernetes.io/projected/05b8994a-b250-43d2-9b62-339fd6089542-kube-api-access-8pwkw\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:14.688295 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.688251    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05b8994a-b250-43d2-9b62-339fd6089542-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:14.688462 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.688434    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9ec146-213e-4f76-b053-2514577ca7bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d9ec146-213e-4f76-b053-2514577ca7bc" (UID: "5d9ec146-213e-4f76-b053-2514577ca7bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:00:14.690273 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.690254    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9ec146-213e-4f76-b053-2514577ca7bc-kube-api-access-7tkmz" (OuterVolumeSpecName: "kube-api-access-7tkmz") pod "5d9ec146-213e-4f76-b053-2514577ca7bc" (UID: "5d9ec146-213e-4f76-b053-2514577ca7bc"). InnerVolumeSpecName "kube-api-access-7tkmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:00:14.789549 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.789515    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ec146-213e-4f76-b053-2514577ca7bc-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:14.789549 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:14.789545    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tkmz\" (UniqueName: \"kubernetes.io/projected/5d9ec146-213e-4f76-b053-2514577ca7bc-kube-api-access-7tkmz\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:15.016947 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.016861    2555 generic.go:356] "Generic (PLEG): container finished" podID="05b8994a-b250-43d2-9b62-339fd6089542" containerID="420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7" exitCode=0
Jul 29 20:00:15.016947 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.016916    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"
Jul 29 20:00:15.017162 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.016945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852" event={"ID":"05b8994a-b250-43d2-9b62-339fd6089542","Type":"ContainerDied","Data":"420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7"}
Jul 29 20:00:15.017162 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.017011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852" event={"ID":"05b8994a-b250-43d2-9b62-339fd6089542","Type":"ContainerDied","Data":"17f13d231773a5aac5628bfb628b6f44753af5b7c7d0410350ea6ec8bf124154"}
Jul 29 20:00:15.017162 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.017058    2555 scope.go:117] "RemoveContainer" containerID="420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7"
Jul 29 20:00:15.018441 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.018416    2555 generic.go:356] "Generic (PLEG): container finished" podID="5d9ec146-213e-4f76-b053-2514577ca7bc" containerID="e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d" exitCode=0
Jul 29 20:00:15.018736 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.018474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b" event={"ID":"5d9ec146-213e-4f76-b053-2514577ca7bc","Type":"ContainerDied","Data":"e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d"}
Jul 29 20:00:15.018736 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.018520    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"
Jul 29 20:00:15.018736 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.018523    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b" event={"ID":"5d9ec146-213e-4f76-b053-2514577ca7bc","Type":"ContainerDied","Data":"87df45451f849e72a9d08ba28758887a740634613116b3e69fbd3451391d3cef"}
Jul 29 20:00:15.027272 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.027246    2555 scope.go:117] "RemoveContainer" containerID="420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7"
Jul 29 20:00:15.027697 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:15.027554    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7\": container with ID starting with 420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7 not found: ID does not exist" containerID="420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7"
Jul 29 20:00:15.027697 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.027619    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7"} err="failed to get container status \"420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7\": rpc error: code = NotFound desc = could not find container \"420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7\": container with ID starting with 420427c8e685ec8e2410e7396c5b1709b05487bcc3dc47ec5854754cc5626bd7 not found: ID does not exist"
Jul 29 20:00:15.027697 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.027653    2555 scope.go:117] "RemoveContainer" containerID="e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d"
Jul 29 20:00:15.035866 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.035850    2555 scope.go:117] "RemoveContainer" containerID="e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d"
Jul 29 20:00:15.036127 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:15.036108    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d\": container with ID starting with e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d not found: ID does not exist" containerID="e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d"
Jul 29 20:00:15.036193 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.036133    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d"} err="failed to get container status \"e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d\": rpc error: code = NotFound desc = could not find container \"e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d\": container with ID starting with e38aa85831d483b4f8db37f14376d6ac2e4bf9f0cd3dba01f745a46265e4ab1d not found: ID does not exist"
Jul 29 20:00:15.042885 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.042864    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"]
Jul 29 20:00:15.048122 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.048080    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-73e856f2-7bb8775868-98852"]
Jul 29 20:00:15.066520 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.066493    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"]
Jul 29 20:00:15.070412 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:15.070388    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e856f2-68c495b7d8-97s2b"]
Jul 29 20:00:16.658678 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:16.658649    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05b8994a-b250-43d2-9b62-339fd6089542" path="/var/lib/kubelet/pods/05b8994a-b250-43d2-9b62-339fd6089542/volumes"
Jul 29 20:00:16.659049 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:16.659010    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9ec146-213e-4f76-b053-2514577ca7bc" path="/var/lib/kubelet/pods/5d9ec146-213e-4f76-b053-2514577ca7bc/volumes"
Jul 29 20:00:18.776654 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.776617    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"]
Jul 29 20:00:18.777624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.777281    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9ec146-213e-4f76-b053-2514577ca7bc" containerName="payload-processing"
Jul 29 20:00:18.777624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.777305    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9ec146-213e-4f76-b053-2514577ca7bc" containerName="payload-processing"
Jul 29 20:00:18.777624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.777353    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b8994a-b250-43d2-9b62-339fd6089542" containerName="maas-api"
Jul 29 20:00:18.777624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.777362    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b8994a-b250-43d2-9b62-339fd6089542" containerName="maas-api"
Jul 29 20:00:18.777624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.777449    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9ec146-213e-4f76-b053-2514577ca7bc" containerName="payload-processing"
Jul 29 20:00:18.777624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.777465    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b8994a-b250-43d2-9b62-339fd6089542" containerName="maas-api"
Jul 29 20:00:18.782718 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.782684    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"]
Jul 29 20:00:18.782844 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.782824    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.785350 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.785329    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2e1a0620-serving-cert\""
Jul 29 20:00:18.785528 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.785510    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 20:00:18.825669 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.825639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564c86cd-15d2-4426-b013-7dfa48cb86e8-openshift-service-ca\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.825819 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.825691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.825819 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.825722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxwrb\" (UniqueName: \"kubernetes.io/projected/564c86cd-15d2-4426-b013-7dfa48cb86e8-kube-api-access-bxwrb\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.926893 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.926853    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564c86cd-15d2-4426-b013-7dfa48cb86e8-openshift-service-ca\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.927096 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.926918    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.927096 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.926949    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxwrb\" (UniqueName: \"kubernetes.io/projected/564c86cd-15d2-4426-b013-7dfa48cb86e8-kube-api-access-bxwrb\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.927096 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:18.927033    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-serving-cert: secret "maas-api-e2e-derive-2e1a0620-serving-cert" not found
Jul 29 20:00:18.927269 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:18.927118    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls podName:564c86cd-15d2-4426-b013-7dfa48cb86e8 nodeName:}" failed. No retries permitted until 2026-07-29 20:00:19.4270943 +0000 UTC m=+1825.385589174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls") pod "maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" (UID: "564c86cd-15d2-4426-b013-7dfa48cb86e8") : secret "maas-api-e2e-derive-2e1a0620-serving-cert" not found
Jul 29 20:00:18.927820 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.927789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564c86cd-15d2-4426-b013-7dfa48cb86e8-openshift-service-ca\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:18.937327 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:18.937297    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxwrb\" (UniqueName: \"kubernetes.io/projected/564c86cd-15d2-4426-b013-7dfa48cb86e8-kube-api-access-bxwrb\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:19.430140 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.430099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:19.433378 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.433343    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls\") pod \"maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:19.590489 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.590452    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"]
Jul 29 20:00:19.594105 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.594083    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.595908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.595882    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2e1a0620-dockercfg-zgn64\""
Jul 29 20:00:19.596015 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.595913    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2e1a0620\""
Jul 29 20:00:19.605458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.605433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"]
Jul 29 20:00:19.612618 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.612592    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"]
Jul 29 20:00:19.616539 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.616517    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.626758 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.626732    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"]
Jul 29 20:00:19.631087 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.631064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89sh\" (UniqueName: \"kubernetes.io/projected/1fcd0095-b219-4c96-bb1c-4606b6655c93-kube-api-access-c89sh\") pod \"payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.631197 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.631131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") " pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.631197 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.631176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcd0095-b219-4c96-bb1c-4606b6655c93-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.631311 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.631201    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48z64\" (UniqueName: \"kubernetes.io/projected/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-kube-api-access-48z64\") pod \"payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") " pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.697035 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.696998    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:19.732295 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.732259    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89sh\" (UniqueName: \"kubernetes.io/projected/1fcd0095-b219-4c96-bb1c-4606b6655c93-kube-api-access-c89sh\") pod \"payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.732468 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.732356    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") " pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.732468 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.732406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcd0095-b219-4c96-bb1c-4606b6655c93-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.732468 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.732430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48z64\" (UniqueName: \"kubernetes.io/projected/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-kube-api-access-48z64\") pod \"payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") " pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.733249 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.733222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcd0095-b219-4c96-bb1c-4606b6655c93-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.733413 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.733391    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-plugins-config-volume\") pod \"payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") " pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.741725 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.741639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89sh\" (UniqueName: \"kubernetes.io/projected/1fcd0095-b219-4c96-bb1c-4606b6655c93-kube-api-access-c89sh\") pod \"payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.743141 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.743112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48z64\" (UniqueName: \"kubernetes.io/projected/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-kube-api-access-48z64\") pod \"payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") " pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:19.842351 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.842319    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"]
Jul 29 20:00:19.844987 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:00:19.844951    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564c86cd_15d2_4426_b013_7dfa48cb86e8.slice/crio-4bc1a9f90d5c8091d845d55fb7f1f6cf5b3e4e8265086a1d7ecf546e6f18dbbf WatchSource:0}: Error finding container 4bc1a9f90d5c8091d845d55fb7f1f6cf5b3e4e8265086a1d7ecf546e6f18dbbf: Status 404 returned error can't find the container with id 4bc1a9f90d5c8091d845d55fb7f1f6cf5b3e4e8265086a1d7ecf546e6f18dbbf
Jul 29 20:00:19.905528 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.905503    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:19.927864 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:19.927836    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:20.048701 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:20.048512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" event={"ID":"564c86cd-15d2-4426-b013-7dfa48cb86e8","Type":"ContainerStarted","Data":"4bc1a9f90d5c8091d845d55fb7f1f6cf5b3e4e8265086a1d7ecf546e6f18dbbf"}
Jul 29 20:00:20.094479 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:20.094433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"]
Jul 29 20:00:20.096952 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:00:20.096924    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6d46b_123c_45f0_ba91_155d5bf1df9a.slice/crio-55e097133541e965bc3004218a091deef4039caa0be862d8cef630c39c3263d1 WatchSource:0}: Error finding container 55e097133541e965bc3004218a091deef4039caa0be862d8cef630c39c3263d1: Status 404 returned error can't find the container with id 55e097133541e965bc3004218a091deef4039caa0be862d8cef630c39c3263d1
Jul 29 20:00:20.266262 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:20.266235    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"]
Jul 29 20:00:20.269209 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:00:20.269181    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcd0095_b219_4c96_bb1c_4606b6655c93.slice/crio-92a44036255f946dcc6924d036562f593c89c1ae6ad977a527bbd4c3bf49e6c7 WatchSource:0}: Error finding container 92a44036255f946dcc6924d036562f593c89c1ae6ad977a527bbd4c3bf49e6c7: Status 404 returned error can't find the container with id 92a44036255f946dcc6924d036562f593c89c1ae6ad977a527bbd4c3bf49e6c7
Jul 29 20:00:21.058437 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.058362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm" event={"ID":"d4d6d46b-123c-45f0-ba91-155d5bf1df9a","Type":"ContainerStarted","Data":"1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea"}
Jul 29 20:00:21.058437 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.058434    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm" event={"ID":"d4d6d46b-123c-45f0-ba91-155d5bf1df9a","Type":"ContainerStarted","Data":"55e097133541e965bc3004218a091deef4039caa0be862d8cef630c39c3263d1"}
Jul 29 20:00:21.058983 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.058845    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:21.060418 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.060390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2" event={"ID":"1fcd0095-b219-4c96-bb1c-4606b6655c93","Type":"ContainerStarted","Data":"047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792"}
Jul 29 20:00:21.060616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.060445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2" event={"ID":"1fcd0095-b219-4c96-bb1c-4606b6655c93","Type":"ContainerStarted","Data":"92a44036255f946dcc6924d036562f593c89c1ae6ad977a527bbd4c3bf49e6c7"}
Jul 29 20:00:21.060616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.060477    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:21.062168 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.062138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" event={"ID":"564c86cd-15d2-4426-b013-7dfa48cb86e8","Type":"ContainerStarted","Data":"3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa"}
Jul 29 20:00:21.062297 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.062256    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:21.085238 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.085179    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm" podStartSLOduration=2.085160286 podStartE2EDuration="2.085160286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:21.083594386 +0000 UTC m=+1827.042089301" watchObservedRunningTime="2026-07-29 20:00:21.085160286 +0000 UTC m=+1827.043655181"
Jul 29 20:00:21.101299 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.101246    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" podStartSLOduration=2.536241484 podStartE2EDuration="3.101228809s" totalImagesPullingTime="564.987325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:21.099254284 +0000 UTC m=+1827.057749203" watchObservedRunningTime="2026-07-29 20:00:21.101228809 +0000 UTC m=+1827.059723754"
Jul 29 20:00:21.114681 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:21.114621    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2" podStartSLOduration=2.114598922 podStartE2EDuration="2.114598922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:00:21.113186901 +0000 UTC m=+1827.071681821" watchObservedRunningTime="2026-07-29 20:00:21.114598922 +0000 UTC m=+1827.073093817"
Jul 29 20:00:27.073034 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:27.072996    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:32.069760 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:32.069726    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:32.070171 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:32.069816    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:49.530741 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.530706    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"]
Jul 29 20:00:49.531263 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.531019    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" podUID="564c86cd-15d2-4426-b013-7dfa48cb86e8" containerName="maas-api" containerID="cri-o://3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa" gracePeriod=30
Jul 29 20:00:49.597066 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.597033    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"]
Jul 29 20:00:49.597319 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.597270    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm" podUID="d4d6d46b-123c-45f0-ba91-155d5bf1df9a" containerName="payload-processing" containerID="cri-o://1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea" gracePeriod=30
Jul 29 20:00:49.605943 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.605913    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"]
Jul 29 20:00:49.606239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.606213    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2" podUID="1fcd0095-b219-4c96-bb1c-4606b6655c93" containerName="payload-pre-processing" containerID="cri-o://047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792" gracePeriod=30
Jul 29 20:00:49.609186 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:49.608523    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6d46b_123c_45f0_ba91_155d5bf1df9a.slice/crio-1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea.scope\": RecentStats: unable to find data in memory cache]"
Jul 29 20:00:49.889984 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.889955    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:49.893682 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.893648    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:49.927075 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:49.926200    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:50.034705 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.034670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxwrb\" (UniqueName: \"kubernetes.io/projected/564c86cd-15d2-4426-b013-7dfa48cb86e8-kube-api-access-bxwrb\") pod \"564c86cd-15d2-4426-b013-7dfa48cb86e8\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") "
Jul 29 20:00:50.034881 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.034731    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcd0095-b219-4c96-bb1c-4606b6655c93-plugins-config-volume\") pod \"1fcd0095-b219-4c96-bb1c-4606b6655c93\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") "
Jul 29 20:00:50.034881 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.034757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-plugins-config-volume\") pod \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") "
Jul 29 20:00:50.034971 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.034884    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564c86cd-15d2-4426-b013-7dfa48cb86e8-openshift-service-ca\") pod \"564c86cd-15d2-4426-b013-7dfa48cb86e8\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") "
Jul 29 20:00:50.035013 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.034972    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls\") pod \"564c86cd-15d2-4426-b013-7dfa48cb86e8\" (UID: \"564c86cd-15d2-4426-b013-7dfa48cb86e8\") "
Jul 29 20:00:50.035013 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035000    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c89sh\" (UniqueName: \"kubernetes.io/projected/1fcd0095-b219-4c96-bb1c-4606b6655c93-kube-api-access-c89sh\") pod \"1fcd0095-b219-4c96-bb1c-4606b6655c93\" (UID: \"1fcd0095-b219-4c96-bb1c-4606b6655c93\") "
Jul 29 20:00:50.035117 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035027    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48z64\" (UniqueName: \"kubernetes.io/projected/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-kube-api-access-48z64\") pod \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\" (UID: \"d4d6d46b-123c-45f0-ba91-155d5bf1df9a\") "
Jul 29 20:00:50.035280 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035242    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcd0095-b219-4c96-bb1c-4606b6655c93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fcd0095-b219-4c96-bb1c-4606b6655c93" (UID: "1fcd0095-b219-4c96-bb1c-4606b6655c93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:00:50.035280 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564c86cd-15d2-4426-b013-7dfa48cb86e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "564c86cd-15d2-4426-b013-7dfa48cb86e8" (UID: "564c86cd-15d2-4426-b013-7dfa48cb86e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:00:50.035280 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4d6d46b-123c-45f0-ba91-155d5bf1df9a" (UID: "d4d6d46b-123c-45f0-ba91-155d5bf1df9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:00:50.035501 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035371    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcd0095-b219-4c96-bb1c-4606b6655c93-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.035501 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035392    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.035501 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.035407    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/564c86cd-15d2-4426-b013-7dfa48cb86e8-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.037277 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.037247    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "564c86cd-15d2-4426-b013-7dfa48cb86e8" (UID: "564c86cd-15d2-4426-b013-7dfa48cb86e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:00:50.037405 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.037388    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-kube-api-access-48z64" (OuterVolumeSpecName: "kube-api-access-48z64") pod "d4d6d46b-123c-45f0-ba91-155d5bf1df9a" (UID: "d4d6d46b-123c-45f0-ba91-155d5bf1df9a"). InnerVolumeSpecName "kube-api-access-48z64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:00:50.037702 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.037683    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564c86cd-15d2-4426-b013-7dfa48cb86e8-kube-api-access-bxwrb" (OuterVolumeSpecName: "kube-api-access-bxwrb") pod "564c86cd-15d2-4426-b013-7dfa48cb86e8" (UID: "564c86cd-15d2-4426-b013-7dfa48cb86e8"). InnerVolumeSpecName "kube-api-access-bxwrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:00:50.037761 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.037697    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcd0095-b219-4c96-bb1c-4606b6655c93-kube-api-access-c89sh" (OuterVolumeSpecName: "kube-api-access-c89sh") pod "1fcd0095-b219-4c96-bb1c-4606b6655c93" (UID: "1fcd0095-b219-4c96-bb1c-4606b6655c93"). InnerVolumeSpecName "kube-api-access-c89sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:00:50.136118 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.136043    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/564c86cd-15d2-4426-b013-7dfa48cb86e8-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.136118 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.136071    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c89sh\" (UniqueName: \"kubernetes.io/projected/1fcd0095-b219-4c96-bb1c-4606b6655c93-kube-api-access-c89sh\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.136118 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.136080    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48z64\" (UniqueName: \"kubernetes.io/projected/d4d6d46b-123c-45f0-ba91-155d5bf1df9a-kube-api-access-48z64\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.136118 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.136089    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxwrb\" (UniqueName: \"kubernetes.io/projected/564c86cd-15d2-4426-b013-7dfa48cb86e8-kube-api-access-bxwrb\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:00:50.205319 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.205282    2555 generic.go:356] "Generic (PLEG): container finished" podID="1fcd0095-b219-4c96-bb1c-4606b6655c93" containerID="047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792" exitCode=0
Jul 29 20:00:50.205514 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.205331    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"
Jul 29 20:00:50.205514 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.205373    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2" event={"ID":"1fcd0095-b219-4c96-bb1c-4606b6655c93","Type":"ContainerDied","Data":"047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792"}
Jul 29 20:00:50.205514 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.205464    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2" event={"ID":"1fcd0095-b219-4c96-bb1c-4606b6655c93","Type":"ContainerDied","Data":"92a44036255f946dcc6924d036562f593c89c1ae6ad977a527bbd4c3bf49e6c7"}
Jul 29 20:00:50.205514 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.205501    2555 scope.go:117] "RemoveContainer" containerID="047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792"
Jul 29 20:00:50.206711 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.206690    2555 generic.go:356] "Generic (PLEG): container finished" podID="564c86cd-15d2-4426-b013-7dfa48cb86e8" containerID="3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa" exitCode=0
Jul 29 20:00:50.206813 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.206738    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"
Jul 29 20:00:50.206813 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.206774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" event={"ID":"564c86cd-15d2-4426-b013-7dfa48cb86e8","Type":"ContainerDied","Data":"3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa"}
Jul 29 20:00:50.206919 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.206824    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l" event={"ID":"564c86cd-15d2-4426-b013-7dfa48cb86e8","Type":"ContainerDied","Data":"4bc1a9f90d5c8091d845d55fb7f1f6cf5b3e4e8265086a1d7ecf546e6f18dbbf"}
Jul 29 20:00:50.208203 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.208178    2555 generic.go:356] "Generic (PLEG): container finished" podID="d4d6d46b-123c-45f0-ba91-155d5bf1df9a" containerID="1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea" exitCode=0
Jul 29 20:00:50.208326 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.208211    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"
Jul 29 20:00:50.208326 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.208242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm" event={"ID":"d4d6d46b-123c-45f0-ba91-155d5bf1df9a","Type":"ContainerDied","Data":"1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea"}
Jul 29 20:00:50.208326 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.208281    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm" event={"ID":"d4d6d46b-123c-45f0-ba91-155d5bf1df9a","Type":"ContainerDied","Data":"55e097133541e965bc3004218a091deef4039caa0be862d8cef630c39c3263d1"}
Jul 29 20:00:50.215720 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.215653    2555 scope.go:117] "RemoveContainer" containerID="047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792"
Jul 29 20:00:50.216037 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:50.216013    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792\": container with ID starting with 047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792 not found: ID does not exist" containerID="047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792"
Jul 29 20:00:50.216104 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.216044    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792"} err="failed to get container status \"047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792\": rpc error: code = NotFound desc = could not find container \"047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792\": container with ID starting with 047c6f7cef7cafbcf81c7810496ebebe2986a74b4ff325f85206f59177dde792 not found: ID does not exist"
Jul 29 20:00:50.216104 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.216060    2555 scope.go:117] "RemoveContainer" containerID="3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa"
Jul 29 20:00:50.224442 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.224427    2555 scope.go:117] "RemoveContainer" containerID="3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa"
Jul 29 20:00:50.224756 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:50.224735    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa\": container with ID starting with 3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa not found: ID does not exist" containerID="3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa"
Jul 29 20:00:50.224808 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.224763    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa"} err="failed to get container status \"3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa\": rpc error: code = NotFound desc = could not find container \"3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa\": container with ID starting with 3000521d8e5b754f41e1402321566045d32f53af4c6ab6734d4941c19738c2aa not found: ID does not exist"
Jul 29 20:00:50.224808 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.224779    2555 scope.go:117] "RemoveContainer" containerID="1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea"
Jul 29 20:00:50.232431 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.232356    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"]
Jul 29 20:00:50.232491 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.232438    2555 scope.go:117] "RemoveContainer" containerID="1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea"
Jul 29 20:00:50.232777 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:00:50.232759    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea\": container with ID starting with 1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea not found: ID does not exist" containerID="1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea"
Jul 29 20:00:50.232850 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.232782    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea"} err="failed to get container status \"1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea\": rpc error: code = NotFound desc = could not find container \"1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea\": container with ID starting with 1f51a0a50c78f91b92274d20db562d0d489192842fe8ea0981face7c080fccea not found: ID does not exist"
Jul 29 20:00:50.237405 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.237373    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2e1a0620-6b67466f7b-bpnt2"]
Jul 29 20:00:50.247263 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.247241    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"]
Jul 29 20:00:50.251737 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.251716    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2e1a0620-749c58bfbb-qb4pm"]
Jul 29 20:00:50.263070 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.263046    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"]
Jul 29 20:00:50.267004 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.266984    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e1a0620-c8f64d86-r5p6l"]
Jul 29 20:00:50.659168 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.659126    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcd0095-b219-4c96-bb1c-4606b6655c93" path="/var/lib/kubelet/pods/1fcd0095-b219-4c96-bb1c-4606b6655c93/volumes"
Jul 29 20:00:50.659733 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.659714    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564c86cd-15d2-4426-b013-7dfa48cb86e8" path="/var/lib/kubelet/pods/564c86cd-15d2-4426-b013-7dfa48cb86e8/volumes"
Jul 29 20:00:50.660103 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:00:50.660086    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d6d46b-123c-45f0-ba91-155d5bf1df9a" path="/var/lib/kubelet/pods/d4d6d46b-123c-45f0-ba91-155d5bf1df9a/volumes"
Jul 29 20:02:42.045884 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.045848    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"]
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046229    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcd0095-b219-4c96-bb1c-4606b6655c93" containerName="payload-pre-processing"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046240    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcd0095-b219-4c96-bb1c-4606b6655c93" containerName="payload-pre-processing"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046261    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d6d46b-123c-45f0-ba91-155d5bf1df9a" containerName="payload-processing"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046267    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d6d46b-123c-45f0-ba91-155d5bf1df9a" containerName="payload-processing"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046273    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564c86cd-15d2-4426-b013-7dfa48cb86e8" containerName="maas-api"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046278    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="564c86cd-15d2-4426-b013-7dfa48cb86e8" containerName="maas-api"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046343    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcd0095-b219-4c96-bb1c-4606b6655c93" containerName="payload-pre-processing"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046356    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d6d46b-123c-45f0-ba91-155d5bf1df9a" containerName="payload-processing"
Jul 29 20:02:42.046471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.046364    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="564c86cd-15d2-4426-b013-7dfa48cb86e8" containerName="maas-api"
Jul 29 20:02:42.049336 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.049316    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.051272 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.051251    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f42b0d30-serving-cert\""
Jul 29 20:02:42.051409 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.051369    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 20:02:42.060918 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.060894    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"]
Jul 29 20:02:42.133770 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.133733    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fa9a66-050c-4c96-b977-8e772586f2be-openshift-service-ca\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.133952 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.133785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fa9a66-050c-4c96-b977-8e772586f2be-maas-api-tls\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.133952 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.133844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8pf\" (UniqueName: \"kubernetes.io/projected/28fa9a66-050c-4c96-b977-8e772586f2be-kube-api-access-tz8pf\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.234404 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.234368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fa9a66-050c-4c96-b977-8e772586f2be-openshift-service-ca\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.234610 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.234411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fa9a66-050c-4c96-b977-8e772586f2be-maas-api-tls\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.234610 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.234451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8pf\" (UniqueName: \"kubernetes.io/projected/28fa9a66-050c-4c96-b977-8e772586f2be-kube-api-access-tz8pf\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.235181 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.235144    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fa9a66-050c-4c96-b977-8e772586f2be-openshift-service-ca\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.237491 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.237467    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fa9a66-050c-4c96-b977-8e772586f2be-maas-api-tls\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.245545 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.245518    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8pf\" (UniqueName: \"kubernetes.io/projected/28fa9a66-050c-4c96-b977-8e772586f2be-kube-api-access-tz8pf\") pod \"maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.361534 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.361443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:42.501342 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.501315    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"]
Jul 29 20:02:42.503917 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:02:42.503876    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fa9a66_050c_4c96_b977_8e772586f2be.slice/crio-dd28e29cd4216e4bf32e043ff60a7c3e93e9a27c747af0350f2e6b59e78e5876 WatchSource:0}: Error finding container dd28e29cd4216e4bf32e043ff60a7c3e93e9a27c747af0350f2e6b59e78e5876: Status 404 returned error can't find the container with id dd28e29cd4216e4bf32e043ff60a7c3e93e9a27c747af0350f2e6b59e78e5876
Jul 29 20:02:42.740812 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.740771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx" event={"ID":"28fa9a66-050c-4c96-b977-8e772586f2be","Type":"ContainerStarted","Data":"dd28e29cd4216e4bf32e043ff60a7c3e93e9a27c747af0350f2e6b59e78e5876"}
Jul 29 20:02:42.857168 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.857131    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"]
Jul 29 20:02:42.861737 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.861719    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:42.863501 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.863478    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f42b0d30-dockercfg-69xfx\""
Jul 29 20:02:42.863704 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.863500    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f42b0d30\""
Jul 29 20:02:42.870040 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.870014    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"]
Jul 29 20:02:42.940417 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.940383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c167911-7737-426d-afc3-550bf9393c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:42.940613 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:42.940449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nqz\" (UniqueName: \"kubernetes.io/projected/3c167911-7737-426d-afc3-550bf9393c98-kube-api-access-75nqz\") pod \"payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.041798 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.041773    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c167911-7737-426d-afc3-550bf9393c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.041918 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.041829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nqz\" (UniqueName: \"kubernetes.io/projected/3c167911-7737-426d-afc3-550bf9393c98-kube-api-access-75nqz\") pod \"payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.042449 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.042425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c167911-7737-426d-afc3-550bf9393c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.052948 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.052844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nqz\" (UniqueName: \"kubernetes.io/projected/3c167911-7737-426d-afc3-550bf9393c98-kube-api-access-75nqz\") pod \"payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.173261 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.173220    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.320398 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.320372    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"]
Jul 29 20:02:43.322786 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:02:43.322747    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c167911_7737_426d_afc3_550bf9393c98.slice/crio-f959259cbd92cde769c341ffde821d89818d727785ba44dc818f44e44a9941b6 WatchSource:0}: Error finding container f959259cbd92cde769c341ffde821d89818d727785ba44dc818f44e44a9941b6: Status 404 returned error can't find the container with id f959259cbd92cde769c341ffde821d89818d727785ba44dc818f44e44a9941b6
Jul 29 20:02:43.748207 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.748161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx" event={"ID":"28fa9a66-050c-4c96-b977-8e772586f2be","Type":"ContainerStarted","Data":"1bb82b8f410781ba5896db4100e1f455a31e8e6f35a6b80b651af81749422963"}
Jul 29 20:02:43.748406 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.748258    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:43.749951 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.749925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf" event={"ID":"3c167911-7737-426d-afc3-550bf9393c98","Type":"ContainerStarted","Data":"f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861"}
Jul 29 20:02:43.750078 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.749965    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf" event={"ID":"3c167911-7737-426d-afc3-550bf9393c98","Type":"ContainerStarted","Data":"f959259cbd92cde769c341ffde821d89818d727785ba44dc818f44e44a9941b6"}
Jul 29 20:02:43.750078 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.750010    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:02:43.776086 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.776031    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx" podStartSLOduration=1.260151987 podStartE2EDuration="1.776014082s" totalImagesPullingTime="515.862095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:02:43.773474374 +0000 UTC m=+1969.731969269" watchObservedRunningTime="2026-07-29 20:02:43.776014082 +0000 UTC m=+1969.734508976"
Jul 29 20:02:43.796362 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:43.796314    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf" podStartSLOduration=1.796298938 podStartE2EDuration="1.796298938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:02:43.793790791 +0000 UTC m=+1969.752285721" watchObservedRunningTime="2026-07-29 20:02:43.796298938 +0000 UTC m=+1969.754793832"
Jul 29 20:02:49.760224 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:49.760189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:02:54.757099 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:02:54.757068    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:03:24.827508 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:24.827471    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"]
Jul 29 20:03:24.828091 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:24.827750    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx" podUID="28fa9a66-050c-4c96-b977-8e772586f2be" containerName="maas-api" containerID="cri-o://1bb82b8f410781ba5896db4100e1f455a31e8e6f35a6b80b651af81749422963" gracePeriod=30
Jul 29 20:03:24.900249 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:24.900212    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"]
Jul 29 20:03:24.900483 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:24.900454    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf" podUID="3c167911-7737-426d-afc3-550bf9393c98" containerName="payload-pre-processing" containerID="cri-o://f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861" gracePeriod=30
Jul 29 20:03:24.948384 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:24.948351    2555 generic.go:356] "Generic (PLEG): container finished" podID="28fa9a66-050c-4c96-b977-8e772586f2be" containerID="1bb82b8f410781ba5896db4100e1f455a31e8e6f35a6b80b651af81749422963" exitCode=0
Jul 29 20:03:24.948556 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:24.948411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx" event={"ID":"28fa9a66-050c-4c96-b977-8e772586f2be","Type":"ContainerDied","Data":"1bb82b8f410781ba5896db4100e1f455a31e8e6f35a6b80b651af81749422963"}
Jul 29 20:03:25.146298 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.146273    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:03:25.184206 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.184179    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:03:25.225944 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.225912    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c167911-7737-426d-afc3-550bf9393c98-plugins-config-volume\") pod \"3c167911-7737-426d-afc3-550bf9393c98\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") "
Jul 29 20:03:25.226107 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.225963    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz8pf\" (UniqueName: \"kubernetes.io/projected/28fa9a66-050c-4c96-b977-8e772586f2be-kube-api-access-tz8pf\") pod \"28fa9a66-050c-4c96-b977-8e772586f2be\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") "
Jul 29 20:03:25.226107 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.225983    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fa9a66-050c-4c96-b977-8e772586f2be-openshift-service-ca\") pod \"28fa9a66-050c-4c96-b977-8e772586f2be\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") "
Jul 29 20:03:25.226107 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.226031    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75nqz\" (UniqueName: \"kubernetes.io/projected/3c167911-7737-426d-afc3-550bf9393c98-kube-api-access-75nqz\") pod \"3c167911-7737-426d-afc3-550bf9393c98\" (UID: \"3c167911-7737-426d-afc3-550bf9393c98\") "
Jul 29 20:03:25.226107 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.226055    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fa9a66-050c-4c96-b977-8e772586f2be-maas-api-tls\") pod \"28fa9a66-050c-4c96-b977-8e772586f2be\" (UID: \"28fa9a66-050c-4c96-b977-8e772586f2be\") "
Jul 29 20:03:25.226417 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.226380    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c167911-7737-426d-afc3-550bf9393c98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c167911-7737-426d-afc3-550bf9393c98" (UID: "3c167911-7737-426d-afc3-550bf9393c98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:03:25.226417 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.226398    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fa9a66-050c-4c96-b977-8e772586f2be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28fa9a66-050c-4c96-b977-8e772586f2be" (UID: "28fa9a66-050c-4c96-b977-8e772586f2be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:03:25.228294 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.228265    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fa9a66-050c-4c96-b977-8e772586f2be-kube-api-access-tz8pf" (OuterVolumeSpecName: "kube-api-access-tz8pf") pod "28fa9a66-050c-4c96-b977-8e772586f2be" (UID: "28fa9a66-050c-4c96-b977-8e772586f2be"). InnerVolumeSpecName "kube-api-access-tz8pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:03:25.228294 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.228271    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fa9a66-050c-4c96-b977-8e772586f2be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28fa9a66-050c-4c96-b977-8e772586f2be" (UID: "28fa9a66-050c-4c96-b977-8e772586f2be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:03:25.228416 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.228293    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c167911-7737-426d-afc3-550bf9393c98-kube-api-access-75nqz" (OuterVolumeSpecName: "kube-api-access-75nqz") pod "3c167911-7737-426d-afc3-550bf9393c98" (UID: "3c167911-7737-426d-afc3-550bf9393c98"). InnerVolumeSpecName "kube-api-access-75nqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:03:25.327384 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.327345    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c167911-7737-426d-afc3-550bf9393c98-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:03:25.327384 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.327376    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8pf\" (UniqueName: \"kubernetes.io/projected/28fa9a66-050c-4c96-b977-8e772586f2be-kube-api-access-tz8pf\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:03:25.327384 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.327385    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fa9a66-050c-4c96-b977-8e772586f2be-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:03:25.327384 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.327394    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75nqz\" (UniqueName: \"kubernetes.io/projected/3c167911-7737-426d-afc3-550bf9393c98-kube-api-access-75nqz\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:03:25.327652 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.327403    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fa9a66-050c-4c96-b977-8e772586f2be-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:03:25.954917 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.954880    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"
Jul 29 20:03:25.954917 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.954894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx" event={"ID":"28fa9a66-050c-4c96-b977-8e772586f2be","Type":"ContainerDied","Data":"dd28e29cd4216e4bf32e043ff60a7c3e93e9a27c747af0350f2e6b59e78e5876"}
Jul 29 20:03:25.955433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.954966    2555 scope.go:117] "RemoveContainer" containerID="1bb82b8f410781ba5896db4100e1f455a31e8e6f35a6b80b651af81749422963"
Jul 29 20:03:25.956478 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.956447    2555 generic.go:356] "Generic (PLEG): container finished" podID="3c167911-7737-426d-afc3-550bf9393c98" containerID="f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861" exitCode=0
Jul 29 20:03:25.956636 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.956484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf" event={"ID":"3c167911-7737-426d-afc3-550bf9393c98","Type":"ContainerDied","Data":"f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861"}
Jul 29 20:03:25.956636 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.956504    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"
Jul 29 20:03:25.956636 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.956540    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf" event={"ID":"3c167911-7737-426d-afc3-550bf9393c98","Type":"ContainerDied","Data":"f959259cbd92cde769c341ffde821d89818d727785ba44dc818f44e44a9941b6"}
Jul 29 20:03:25.964849 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.964779    2555 scope.go:117] "RemoveContainer" containerID="f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861"
Jul 29 20:03:25.973751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.973732    2555 scope.go:117] "RemoveContainer" containerID="f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861"
Jul 29 20:03:25.974010 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:03:25.973988    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861\": container with ID starting with f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861 not found: ID does not exist" containerID="f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861"
Jul 29 20:03:25.974075 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.974023    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861"} err="failed to get container status \"f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861\": rpc error: code = NotFound desc = could not find container \"f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861\": container with ID starting with f5f379c0b86638514c9cfce03b1eb783a9ca366a0dfcd67ee2cd626ca5620861 not found: ID does not exist"
Jul 29 20:03:25.981256 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.981231    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"]
Jul 29 20:03:25.983233 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.983210    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f42b0d30-6c46dc5dd4-84skf"]
Jul 29 20:03:25.993913 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.993889    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"]
Jul 29 20:03:25.998950 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:25.998924    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f42b0d30-85fc6959b6-jjclx"]
Jul 29 20:03:26.659044 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:26.659008    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28fa9a66-050c-4c96-b977-8e772586f2be" path="/var/lib/kubelet/pods/28fa9a66-050c-4c96-b977-8e772586f2be/volumes"
Jul 29 20:03:26.659373 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:26.659360    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c167911-7737-426d-afc3-550bf9393c98" path="/var/lib/kubelet/pods/3c167911-7737-426d-afc3-550bf9393c98/volumes"
Jul 29 20:03:55.399647 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.399604    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"]
Jul 29 20:03:55.400134 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.400017    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c167911-7737-426d-afc3-550bf9393c98" containerName="payload-pre-processing"
Jul 29 20:03:55.400134 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.400029    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c167911-7737-426d-afc3-550bf9393c98" containerName="payload-pre-processing"
Jul 29 20:03:55.400134 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.400050    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28fa9a66-050c-4c96-b977-8e772586f2be" containerName="maas-api"
Jul 29 20:03:55.400134 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.400058    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="28fa9a66-050c-4c96-b977-8e772586f2be" containerName="maas-api"
Jul 29 20:03:55.400134 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.400118    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c167911-7737-426d-afc3-550bf9393c98" containerName="payload-pre-processing"
Jul 29 20:03:55.400134 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.400129    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="28fa9a66-050c-4c96-b977-8e772586f2be" containerName="maas-api"
Jul 29 20:03:55.404893 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.404871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.407080 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.407056    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3a4cdcb0-serving-cert\""
Jul 29 20:03:55.407080 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.407073    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 20:03:55.420201 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.420177    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"]
Jul 29 20:03:55.504036 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.503984    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-maas-api-tls\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.504223 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.504061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g5s8\" (UniqueName: \"kubernetes.io/projected/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-kube-api-access-4g5s8\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.504223 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.504135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-openshift-service-ca\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.605367 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.605323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-openshift-service-ca\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.605566 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.605411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-maas-api-tls\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.605566 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.605459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g5s8\" (UniqueName: \"kubernetes.io/projected/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-kube-api-access-4g5s8\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.606031 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.606009    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-openshift-service-ca\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.608051 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.608028    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-maas-api-tls\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.621208 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.621185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g5s8\" (UniqueName: \"kubernetes.io/projected/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-kube-api-access-4g5s8\") pod \"maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.715503 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.715470    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:55.861547 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.858609    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"]
Jul 29 20:03:55.861547 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:03:55.860197    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c2c9dd_11e3_49f1_b6d0_0dcd87f87824.slice/crio-5171755ba08b29f385472114740623bc783cd213e79dd74e042d3bedbe101da9 WatchSource:0}: Error finding container 5171755ba08b29f385472114740623bc783cd213e79dd74e042d3bedbe101da9: Status 404 returned error can't find the container with id 5171755ba08b29f385472114740623bc783cd213e79dd74e042d3bedbe101da9
Jul 29 20:03:55.862276 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:55.862249    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:03:56.102694 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.102607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg" event={"ID":"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824","Type":"ContainerStarted","Data":"5171755ba08b29f385472114740623bc783cd213e79dd74e042d3bedbe101da9"}
Jul 29 20:03:56.469132 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.469100    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"]
Jul 29 20:03:56.473518 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.473498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.476012 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.475984    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3a4cdcb0\""
Jul 29 20:03:56.476113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.476037    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3a4cdcb0-dockercfg-8btl6\""
Jul 29 20:03:56.484981 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.484956    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"]
Jul 29 20:03:56.581678 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.581635    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"]
Jul 29 20:03:56.585255 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.585239    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.597353 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.597326    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"]
Jul 29 20:03:56.614633 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.614603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mshjg\" (UniqueName: \"kubernetes.io/projected/31b9e6b0-38cd-451e-b0a0-849c479be24d-kube-api-access-mshjg\") pod \"payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.614764 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.614661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b9e6b0-38cd-451e-b0a0-849c479be24d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.715369 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.715275    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9lf\" (UniqueName: \"kubernetes.io/projected/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-kube-api-access-bj9lf\") pod \"payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") " pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.715369 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.715320    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") " pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.715369 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.715364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshjg\" (UniqueName: \"kubernetes.io/projected/31b9e6b0-38cd-451e-b0a0-849c479be24d-kube-api-access-mshjg\") pod \"payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.715725 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.715394    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b9e6b0-38cd-451e-b0a0-849c479be24d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.716107 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.716087    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b9e6b0-38cd-451e-b0a0-849c479be24d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.730767 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.730738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshjg\" (UniqueName: \"kubernetes.io/projected/31b9e6b0-38cd-451e-b0a0-849c479be24d-kube-api-access-mshjg\") pod \"payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.787735 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.787696    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:56.816862 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.816827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9lf\" (UniqueName: \"kubernetes.io/projected/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-kube-api-access-bj9lf\") pod \"payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") " pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.817021 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.816881    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") " pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.817895 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.817866    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") " pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.836519 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.836484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9lf\" (UniqueName: \"kubernetes.io/projected/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-kube-api-access-bj9lf\") pod \"payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") " pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.895535 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.895496    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:03:56.933858 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:56.933824    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"]
Jul 29 20:03:56.935064 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:03:56.935021    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b9e6b0_38cd_451e_b0a0_849c479be24d.slice/crio-65a813d87a525025cd6dda2c77012b20fd8a7736be20008e110be1b27bb54dbb WatchSource:0}: Error finding container 65a813d87a525025cd6dda2c77012b20fd8a7736be20008e110be1b27bb54dbb: Status 404 returned error can't find the container with id 65a813d87a525025cd6dda2c77012b20fd8a7736be20008e110be1b27bb54dbb
Jul 29 20:03:57.040853 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.040733    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"]
Jul 29 20:03:57.043637 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:03:57.043552    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42396b0d_8f20_450f_8e2c_b8e2dcbd1b4d.slice/crio-b46968ab719f683ef5ebfd70873bf46f7776211e6c95c0d7c8c698471ac23de2 WatchSource:0}: Error finding container b46968ab719f683ef5ebfd70873bf46f7776211e6c95c0d7c8c698471ac23de2: Status 404 returned error can't find the container with id b46968ab719f683ef5ebfd70873bf46f7776211e6c95c0d7c8c698471ac23de2
Jul 29 20:03:57.110129 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.110095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh" event={"ID":"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d","Type":"ContainerStarted","Data":"b46968ab719f683ef5ebfd70873bf46f7776211e6c95c0d7c8c698471ac23de2"}
Jul 29 20:03:57.111755 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.111732    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w" event={"ID":"31b9e6b0-38cd-451e-b0a0-849c479be24d","Type":"ContainerStarted","Data":"47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4"}
Jul 29 20:03:57.111882 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.111779    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w" event={"ID":"31b9e6b0-38cd-451e-b0a0-849c479be24d","Type":"ContainerStarted","Data":"65a813d87a525025cd6dda2c77012b20fd8a7736be20008e110be1b27bb54dbb"}
Jul 29 20:03:57.111882 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.111808    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:03:57.113343 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.113319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg" event={"ID":"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824","Type":"ContainerStarted","Data":"cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68"}
Jul 29 20:03:57.113473 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.113458    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:03:57.133198 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.133154    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w" podStartSLOduration=1.133135119 podStartE2EDuration="1.133135119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:03:57.132719925 +0000 UTC m=+2043.091214822" watchObservedRunningTime="2026-07-29 20:03:57.133135119 +0000 UTC m=+2043.091630018"
Jul 29 20:03:57.160120 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:57.160065    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg" podStartSLOduration=1.575998855 podStartE2EDuration="2.160048428s" totalImagesPullingTime="584.049573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:03:57.158365749 +0000 UTC m=+2043.116860663" watchObservedRunningTime="2026-07-29 20:03:57.160048428 +0000 UTC m=+2043.118543322"
Jul 29 20:03:58.119750 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:58.119707    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh" event={"ID":"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d","Type":"ContainerStarted","Data":"08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb"}
Jul 29 20:03:58.140990 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:58.140131    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh" podStartSLOduration=2.140108836 podStartE2EDuration="2.140108836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:03:58.138122853 +0000 UTC m=+2044.096617776" watchObservedRunningTime="2026-07-29 20:03:58.140108836 +0000 UTC m=+2044.098603732"
Jul 29 20:03:59.125085 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:03:59.125049    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:04:03.125289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:03.125252    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:04:08.122091 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:08.122053    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:04:10.131909 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:10.131878    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:04:37.877211 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.877125    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"]
Jul 29 20:04:37.880875 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.880855    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:37.882963 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.882936    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3a4cdcb0\"/\"e2e-lifecycle-model-3a4cdcb0-kserve-self-signed-certs\""
Jul 29 20:04:37.883078 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.883060    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3a4cdcb0\"/\"default-dockercfg-hm74r\""
Jul 29 20:04:37.883135 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.883082    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3a4cdcb0\"/\"kube-root-ca.crt\""
Jul 29 20:04:37.883227 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.883212    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3a4cdcb0\"/\"openshift-service-ca.crt\""
Jul 29 20:04:37.891440 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.891413    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"]
Jul 29 20:04:37.982537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.982502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-dshm\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:37.982724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.982563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ppk\" (UniqueName: \"kubernetes.io/projected/e319458f-1a6e-4f25-881a-4e18c26315a0-kube-api-access-z7ppk\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:37.982724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.982652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-home\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:37.982724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.982690    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e319458f-1a6e-4f25-881a-4e18c26315a0-tls-certs\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:37.982724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.982712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-model-cache\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:37.982868 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:37.982785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-tmp-dir\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.083636 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.083564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-dshm\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.083816 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.083682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ppk\" (UniqueName: \"kubernetes.io/projected/e319458f-1a6e-4f25-881a-4e18c26315a0-kube-api-access-z7ppk\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.083816 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.083754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-home\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.083816 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.083779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e319458f-1a6e-4f25-881a-4e18c26315a0-tls-certs\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.083816 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.083813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-model-cache\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.084046 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.083863    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-tmp-dir\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.084198 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.084170    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-home\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.084272 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.084210    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-model-cache\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.084272 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.084250    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-tmp-dir\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.085985 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.085967    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-dshm\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.086363 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.086341    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e319458f-1a6e-4f25-881a-4e18c26315a0-tls-certs\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.092630 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.092572    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ppk\" (UniqueName: \"kubernetes.io/projected/e319458f-1a6e-4f25-881a-4e18c26315a0-kube-api-access-z7ppk\") pod \"e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") " pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.191723 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.191692    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:38.333112 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:38.333086    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"]
Jul 29 20:04:38.335605 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:04:38.335558    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode319458f_1a6e_4f25_881a_4e18c26315a0.slice/crio-928453431f432a6a219f0c6819500509ba5c298f47887f0d28cab91eb4a3a9bf WatchSource:0}: Error finding container 928453431f432a6a219f0c6819500509ba5c298f47887f0d28cab91eb4a3a9bf: Status 404 returned error can't find the container with id 928453431f432a6a219f0c6819500509ba5c298f47887f0d28cab91eb4a3a9bf
Jul 29 20:04:39.325277 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:39.325237    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk" event={"ID":"e319458f-1a6e-4f25-881a-4e18c26315a0","Type":"ContainerStarted","Data":"ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467"}
Jul 29 20:04:39.325777 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:39.325286    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk" event={"ID":"e319458f-1a6e-4f25-881a-4e18c26315a0","Type":"ContainerStarted","Data":"928453431f432a6a219f0c6819500509ba5c298f47887f0d28cab91eb4a3a9bf"}
Jul 29 20:04:39.344340 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:39.344292    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk" podStartSLOduration=2.34427845 podStartE2EDuration="2.34427845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:04:39.3426938 +0000 UTC m=+2085.301188707" watchObservedRunningTime="2026-07-29 20:04:39.34427845 +0000 UTC m=+2085.302773344"
Jul 29 20:04:48.192175 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:48.192130    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:48.192175 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:48.192180    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:48.204651 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:48.204625    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:48.382806 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:48.382775    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:04:54.655173 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:54.655131    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:04:54.658558 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:04:54.658537    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:05:15.396212 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.396179    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"]
Jul 29 20:05:15.396858 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.396445    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg" podUID="58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" containerName="maas-api" containerID="cri-o://cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68" gracePeriod=30
Jul 29 20:05:15.460880 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.460840    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"]
Jul 29 20:05:15.461258 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.461193    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh" podUID="42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" containerName="payload-processing" containerID="cri-o://08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb" gracePeriod=30
Jul 29 20:05:15.476421 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.476382    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"]
Jul 29 20:05:15.476859 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.476793    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w" podUID="31b9e6b0-38cd-451e-b0a0-849c479be24d" containerName="payload-pre-processing" containerID="cri-o://47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4" gracePeriod=30
Jul 29 20:05:15.747833 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.747806    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:05:15.775269 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.775249    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:05:15.782797 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.782768    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:05:15.839964 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.839931    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-maas-api-tls\") pod \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") "
Jul 29 20:05:15.840146 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.839988    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-openshift-service-ca\") pod \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") "
Jul 29 20:05:15.840146 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.840025    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g5s8\" (UniqueName: \"kubernetes.io/projected/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-kube-api-access-4g5s8\") pod \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\" (UID: \"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824\") "
Jul 29 20:05:15.840378 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.840354    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" (UID: "58c2c9dd-11e3-49f1-b6d0-0dcd87f87824"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:05:15.842343 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.842318    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-kube-api-access-4g5s8" (OuterVolumeSpecName: "kube-api-access-4g5s8") pod "58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" (UID: "58c2c9dd-11e3-49f1-b6d0-0dcd87f87824"). InnerVolumeSpecName "kube-api-access-4g5s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:15.842343 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.842328    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" (UID: "58c2c9dd-11e3-49f1-b6d0-0dcd87f87824"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:05:15.940888 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.940814    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-plugins-config-volume\") pod \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") "
Jul 29 20:05:15.941048 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.940929    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mshjg\" (UniqueName: \"kubernetes.io/projected/31b9e6b0-38cd-451e-b0a0-849c479be24d-kube-api-access-mshjg\") pod \"31b9e6b0-38cd-451e-b0a0-849c479be24d\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") "
Jul 29 20:05:15.941048 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941020    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b9e6b0-38cd-451e-b0a0-849c479be24d-plugins-config-volume\") pod \"31b9e6b0-38cd-451e-b0a0-849c479be24d\" (UID: \"31b9e6b0-38cd-451e-b0a0-849c479be24d\") "
Jul 29 20:05:15.941177 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941050    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj9lf\" (UniqueName: \"kubernetes.io/projected/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-kube-api-access-bj9lf\") pod \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\" (UID: \"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d\") "
Jul 29 20:05:15.941350 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941248    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" (UID: "42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:05:15.941444 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941305    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:15.941444 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941390    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g5s8\" (UniqueName: \"kubernetes.io/projected/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-kube-api-access-4g5s8\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:15.941444 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941421    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:15.941680 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.941446    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b9e6b0-38cd-451e-b0a0-849c479be24d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31b9e6b0-38cd-451e-b0a0-849c479be24d" (UID: "31b9e6b0-38cd-451e-b0a0-849c479be24d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:05:15.943210 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.943185    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b9e6b0-38cd-451e-b0a0-849c479be24d-kube-api-access-mshjg" (OuterVolumeSpecName: "kube-api-access-mshjg") pod "31b9e6b0-38cd-451e-b0a0-849c479be24d" (UID: "31b9e6b0-38cd-451e-b0a0-849c479be24d"). InnerVolumeSpecName "kube-api-access-mshjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:15.943302 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:15.943227    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-kube-api-access-bj9lf" (OuterVolumeSpecName: "kube-api-access-bj9lf") pod "42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" (UID: "42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d"). InnerVolumeSpecName "kube-api-access-bj9lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:16.042600 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.042538    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mshjg\" (UniqueName: \"kubernetes.io/projected/31b9e6b0-38cd-451e-b0a0-849c479be24d-kube-api-access-mshjg\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:16.042600 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.042568    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31b9e6b0-38cd-451e-b0a0-849c479be24d-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:16.042600 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.042605    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj9lf\" (UniqueName: \"kubernetes.io/projected/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-kube-api-access-bj9lf\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:16.042831 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.042620    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:16.509377 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.509342    2555 generic.go:356] "Generic (PLEG): container finished" podID="31b9e6b0-38cd-451e-b0a0-849c479be24d" containerID="47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4" exitCode=0
Jul 29 20:05:16.509806 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.509386    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"
Jul 29 20:05:16.509806 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.509429    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w" event={"ID":"31b9e6b0-38cd-451e-b0a0-849c479be24d","Type":"ContainerDied","Data":"47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4"}
Jul 29 20:05:16.509806 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.509493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w" event={"ID":"31b9e6b0-38cd-451e-b0a0-849c479be24d","Type":"ContainerDied","Data":"65a813d87a525025cd6dda2c77012b20fd8a7736be20008e110be1b27bb54dbb"}
Jul 29 20:05:16.509806 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.509527    2555 scope.go:117] "RemoveContainer" containerID="47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4"
Jul 29 20:05:16.510907 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.510712    2555 generic.go:356] "Generic (PLEG): container finished" podID="58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" containerID="cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68" exitCode=0
Jul 29 20:05:16.510907 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.510744    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"
Jul 29 20:05:16.510907 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.510800    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg" event={"ID":"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824","Type":"ContainerDied","Data":"cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68"}
Jul 29 20:05:16.510907 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.510845    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg" event={"ID":"58c2c9dd-11e3-49f1-b6d0-0dcd87f87824","Type":"ContainerDied","Data":"5171755ba08b29f385472114740623bc783cd213e79dd74e042d3bedbe101da9"}
Jul 29 20:05:16.512365 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.512341    2555 generic.go:356] "Generic (PLEG): container finished" podID="42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" containerID="08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb" exitCode=0
Jul 29 20:05:16.512490 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.512397    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"
Jul 29 20:05:16.512490 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.512427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh" event={"ID":"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d","Type":"ContainerDied","Data":"08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb"}
Jul 29 20:05:16.512594 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.512492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh" event={"ID":"42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d","Type":"ContainerDied","Data":"b46968ab719f683ef5ebfd70873bf46f7776211e6c95c0d7c8c698471ac23de2"}
Jul 29 20:05:16.519046 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.519026    2555 scope.go:117] "RemoveContainer" containerID="47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4"
Jul 29 20:05:16.519317 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:05:16.519297    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4\": container with ID starting with 47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4 not found: ID does not exist" containerID="47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4"
Jul 29 20:05:16.519399 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.519329    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4"} err="failed to get container status \"47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4\": rpc error: code = NotFound desc = could not find container \"47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4\": container with ID starting with 47563343b4f7d2eaa686af765f93e5b9de1487bf4e5f00a3dd299ac61185d2b4 not found: ID does not exist"
Jul 29 20:05:16.519399 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.519354    2555 scope.go:117] "RemoveContainer" containerID="cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68"
Jul 29 20:05:16.528214 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.528199    2555 scope.go:117] "RemoveContainer" containerID="cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68"
Jul 29 20:05:16.528457 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:05:16.528443    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68\": container with ID starting with cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68 not found: ID does not exist" containerID="cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68"
Jul 29 20:05:16.528498 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.528464    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68"} err="failed to get container status \"cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68\": rpc error: code = NotFound desc = could not find container \"cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68\": container with ID starting with cd092a897a4aa78b15a81b08e4af4271c35331b5ad087eb4b2f7b843dc635f68 not found: ID does not exist"
Jul 29 20:05:16.528498 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.528478    2555 scope.go:117] "RemoveContainer" containerID="08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb"
Jul 29 20:05:16.536820 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.536803    2555 scope.go:117] "RemoveContainer" containerID="08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb"
Jul 29 20:05:16.537090 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:05:16.537072    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb\": container with ID starting with 08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb not found: ID does not exist" containerID="08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb"
Jul 29 20:05:16.537151 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.537094    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb"} err="failed to get container status \"08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb\": rpc error: code = NotFound desc = could not find container \"08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb\": container with ID starting with 08a2e1de8eba6c3e53391177fe83929700f5906481b4e15b51119962958db3fb not found: ID does not exist"
Jul 29 20:05:16.537663 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.537645    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"]
Jul 29 20:05:16.542478 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.542458    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a4cdcb0-5f55c4867c-t6gzg"]
Jul 29 20:05:16.552690 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.552668    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"]
Jul 29 20:05:16.555131 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.555111    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a4cdcb0-78c549b57b-zkc9w"]
Jul 29 20:05:16.566420 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.566392    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"]
Jul 29 20:05:16.571372 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.571352    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3a4cdcb0-557b69fff4-kk4gh"]
Jul 29 20:05:16.658384 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.658354    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31b9e6b0-38cd-451e-b0a0-849c479be24d" path="/var/lib/kubelet/pods/31b9e6b0-38cd-451e-b0a0-849c479be24d/volumes"
Jul 29 20:05:16.658754 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.658740    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" path="/var/lib/kubelet/pods/42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d/volumes"
Jul 29 20:05:16.659074 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:16.659062    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" path="/var/lib/kubelet/pods/58c2c9dd-11e3-49f1-b6d0-0dcd87f87824/volumes"
Jul 29 20:05:23.742675 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:23.742632    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"]
Jul 29 20:05:38.762740 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:38.762673    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk" podUID="e319458f-1a6e-4f25-881a-4e18c26315a0" containerName="main" containerID="cri-o://ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467" gracePeriod=45
Jul 29 20:05:39.013941 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.013876    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:05:39.046103 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046075    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-tmp-dir\") pod \"e319458f-1a6e-4f25-881a-4e18c26315a0\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") "
Jul 29 20:05:39.046257 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046128    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-home\") pod \"e319458f-1a6e-4f25-881a-4e18c26315a0\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") "
Jul 29 20:05:39.046257 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046151    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-model-cache\") pod \"e319458f-1a6e-4f25-881a-4e18c26315a0\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") "
Jul 29 20:05:39.046257 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046205    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7ppk\" (UniqueName: \"kubernetes.io/projected/e319458f-1a6e-4f25-881a-4e18c26315a0-kube-api-access-z7ppk\") pod \"e319458f-1a6e-4f25-881a-4e18c26315a0\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") "
Jul 29 20:05:39.046433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046265    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e319458f-1a6e-4f25-881a-4e18c26315a0-tls-certs\") pod \"e319458f-1a6e-4f25-881a-4e18c26315a0\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") "
Jul 29 20:05:39.046433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-dshm\") pod \"e319458f-1a6e-4f25-881a-4e18c26315a0\" (UID: \"e319458f-1a6e-4f25-881a-4e18c26315a0\") "
Jul 29 20:05:39.046433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046355    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e319458f-1a6e-4f25-881a-4e18c26315a0" (UID: "e319458f-1a6e-4f25-881a-4e18c26315a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:39.046609 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046448    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e319458f-1a6e-4f25-881a-4e18c26315a0" (UID: "e319458f-1a6e-4f25-881a-4e18c26315a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:39.046609 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046463    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-home" (OuterVolumeSpecName: "home") pod "e319458f-1a6e-4f25-881a-4e18c26315a0" (UID: "e319458f-1a6e-4f25-881a-4e18c26315a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:39.046741 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046719    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:39.046809 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046747    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:39.046809 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.046761    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:39.048650 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.048613    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e319458f-1a6e-4f25-881a-4e18c26315a0-kube-api-access-z7ppk" (OuterVolumeSpecName: "kube-api-access-z7ppk") pod "e319458f-1a6e-4f25-881a-4e18c26315a0" (UID: "e319458f-1a6e-4f25-881a-4e18c26315a0"). InnerVolumeSpecName "kube-api-access-z7ppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:05:39.049058 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.049038    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-dshm" (OuterVolumeSpecName: "dshm") pod "e319458f-1a6e-4f25-881a-4e18c26315a0" (UID: "e319458f-1a6e-4f25-881a-4e18c26315a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:05:39.049058 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.049050    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e319458f-1a6e-4f25-881a-4e18c26315a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e319458f-1a6e-4f25-881a-4e18c26315a0" (UID: "e319458f-1a6e-4f25-881a-4e18c26315a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:05:39.147709 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.147678    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ppk\" (UniqueName: \"kubernetes.io/projected/e319458f-1a6e-4f25-881a-4e18c26315a0-kube-api-access-z7ppk\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:39.147709 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.147706    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e319458f-1a6e-4f25-881a-4e18c26315a0-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:39.147709 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.147716    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e319458f-1a6e-4f25-881a-4e18c26315a0-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:05:39.638857 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.638818    2555 generic.go:356] "Generic (PLEG): container finished" podID="e319458f-1a6e-4f25-881a-4e18c26315a0" containerID="ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467" exitCode=0
Jul 29 20:05:39.639032 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.638882    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"
Jul 29 20:05:39.639032 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.638898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk" event={"ID":"e319458f-1a6e-4f25-881a-4e18c26315a0","Type":"ContainerDied","Data":"ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467"}
Jul 29 20:05:39.639032 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.638944    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk" event={"ID":"e319458f-1a6e-4f25-881a-4e18c26315a0","Type":"ContainerDied","Data":"928453431f432a6a219f0c6819500509ba5c298f47887f0d28cab91eb4a3a9bf"}
Jul 29 20:05:39.639032 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.638961    2555 scope.go:117] "RemoveContainer" containerID="ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467"
Jul 29 20:05:39.648666 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.648647    2555 scope.go:117] "RemoveContainer" containerID="ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467"
Jul 29 20:05:39.648942 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:05:39.648921    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467\": container with ID starting with ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467 not found: ID does not exist" containerID="ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467"
Jul 29 20:05:39.648996 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.648951    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467"} err="failed to get container status \"ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467\": rpc error: code = NotFound desc = could not find container \"ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467\": container with ID starting with ef16f48b72930d13d577982616347284d3e658e7a5b45d5084d249052253a467 not found: ID does not exist"
Jul 29 20:05:39.660651 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.660624    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"]
Jul 29 20:05:39.666969 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:39.666948    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3a4cdcb0/e2e-lifecycle-model-3a4cdcb0-kserve-7c7955dd55-sjmsk"]
Jul 29 20:05:40.657785 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:05:40.657752    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e319458f-1a6e-4f25-881a-4e18c26315a0" path="/var/lib/kubelet/pods/e319458f-1a6e-4f25-881a-4e18c26315a0/volumes"
Jul 29 20:07:30.418236 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418129    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"]
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418627    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" containerName="payload-processing"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418645    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" containerName="payload-processing"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418656    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" containerName="maas-api"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418661    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" containerName="maas-api"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418676    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31b9e6b0-38cd-451e-b0a0-849c479be24d" containerName="payload-pre-processing"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418681    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="31b9e6b0-38cd-451e-b0a0-849c479be24d" containerName="payload-pre-processing"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418693    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e319458f-1a6e-4f25-881a-4e18c26315a0" containerName="main"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418698    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e319458f-1a6e-4f25-881a-4e18c26315a0" containerName="main"
Jul 29 20:07:30.418751 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418754    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42396b0d-8f20-450f-8e2c-b8e2dcbd1b4d" containerName="payload-processing"
Jul 29 20:07:30.419035 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418763    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c2c9dd-11e3-49f1-b6d0-0dcd87f87824" containerName="maas-api"
Jul 29 20:07:30.419035 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418770    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="31b9e6b0-38cd-451e-b0a0-849c479be24d" containerName="payload-pre-processing"
Jul 29 20:07:30.419035 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.418776    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e319458f-1a6e-4f25-881a-4e18c26315a0" containerName="main"
Jul 29 20:07:30.421724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.421706    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.423871 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.423847    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f5ad4c-serving-cert\""
Jul 29 20:07:30.423981 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.423867    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 20:07:30.431155 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.431130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"]
Jul 29 20:07:30.542223 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.542190    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ffd996-20d7-4e09-84b5-a9e050084100-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.542435 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.542305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ffd996-20d7-4e09-84b5-a9e050084100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.542435 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.542333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rrc\" (UniqueName: \"kubernetes.io/projected/66ffd996-20d7-4e09-84b5-a9e050084100-kube-api-access-99rrc\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.643644 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.643605    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ffd996-20d7-4e09-84b5-a9e050084100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.643644 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.643651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rrc\" (UniqueName: \"kubernetes.io/projected/66ffd996-20d7-4e09-84b5-a9e050084100-kube-api-access-99rrc\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.643905 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.643689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ffd996-20d7-4e09-84b5-a9e050084100-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.644288 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.644255    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ffd996-20d7-4e09-84b5-a9e050084100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.646306 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.646281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ffd996-20d7-4e09-84b5-a9e050084100-maas-api-tls\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.653006 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.652982    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rrc\" (UniqueName: \"kubernetes.io/projected/66ffd996-20d7-4e09-84b5-a9e050084100-kube-api-access-99rrc\") pod \"maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.733419 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.733385    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:30.872432 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:30.872405    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"]
Jul 29 20:07:30.874222 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:07:30.874189    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ffd996_20d7_4e09_84b5_a9e050084100.slice/crio-e165406bf9572b990dc51df8d3b8bc44a34c6ab50c408653cd21097376be69ed WatchSource:0}: Error finding container e165406bf9572b990dc51df8d3b8bc44a34c6ab50c408653cd21097376be69ed: Status 404 returned error can't find the container with id e165406bf9572b990dc51df8d3b8bc44a34c6ab50c408653cd21097376be69ed
Jul 29 20:07:31.131539 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.131452    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"]
Jul 29 20:07:31.135957 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.135934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.137850 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.137827    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f5ad4c\""
Jul 29 20:07:31.143738 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.143710    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"]
Jul 29 20:07:31.162573 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.162516    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz" event={"ID":"66ffd996-20d7-4e09-84b5-a9e050084100","Type":"ContainerStarted","Data":"e165406bf9572b990dc51df8d3b8bc44a34c6ab50c408653cd21097376be69ed"}
Jul 29 20:07:31.163316 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.163295    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"]
Jul 29 20:07:31.166658 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.166642    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.168693 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.168671    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f5ad4c-dockercfg-rnj58\""
Jul 29 20:07:31.177149 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.177121    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"]
Jul 29 20:07:31.248184 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.248151    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a988bfc-5004-4695-9913-a8a269cf2624-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.248184 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.248186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f680b4d-7496-40f5-bb73-d7aeb852e576-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.248417 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.248222    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkp6j\" (UniqueName: \"kubernetes.io/projected/6f680b4d-7496-40f5-bb73-d7aeb852e576-kube-api-access-gkp6j\") pod \"payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.248417 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.248334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gsd\" (UniqueName: \"kubernetes.io/projected/8a988bfc-5004-4695-9913-a8a269cf2624-kube-api-access-q7gsd\") pod \"payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.349535 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.349500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gsd\" (UniqueName: \"kubernetes.io/projected/8a988bfc-5004-4695-9913-a8a269cf2624-kube-api-access-q7gsd\") pod \"payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.349770 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.349564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a988bfc-5004-4695-9913-a8a269cf2624-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.349770 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.349627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f680b4d-7496-40f5-bb73-d7aeb852e576-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.349770 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.349681    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkp6j\" (UniqueName: \"kubernetes.io/projected/6f680b4d-7496-40f5-bb73-d7aeb852e576-kube-api-access-gkp6j\") pod \"payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.350342 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.350316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a988bfc-5004-4695-9913-a8a269cf2624-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.350449 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.350387    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f680b4d-7496-40f5-bb73-d7aeb852e576-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.358616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.358563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gsd\" (UniqueName: \"kubernetes.io/projected/8a988bfc-5004-4695-9913-a8a269cf2624-kube-api-access-q7gsd\") pod \"payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.358756 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.358638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkp6j\" (UniqueName: \"kubernetes.io/projected/6f680b4d-7496-40f5-bb73-d7aeb852e576-kube-api-access-gkp6j\") pod \"payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.448454 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.448421    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:31.478178 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.478152    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:31.599369 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.599338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"]
Jul 29 20:07:31.610041 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:07:31.609990    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f680b4d_7496_40f5_bb73_d7aeb852e576.slice/crio-036a40acb135cc519dd251465aeadd444e61336eef322b93183847c322c3b7b3 WatchSource:0}: Error finding container 036a40acb135cc519dd251465aeadd444e61336eef322b93183847c322c3b7b3: Status 404 returned error can't find the container with id 036a40acb135cc519dd251465aeadd444e61336eef322b93183847c322c3b7b3
Jul 29 20:07:31.635572 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:31.635536    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"]
Jul 29 20:07:31.639481 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:07:31.639452    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a988bfc_5004_4695_9913_a8a269cf2624.slice/crio-811093373bea9fa01ed110d0a5a54dc9aa86c8fb92190a3f34601e32354408ed WatchSource:0}: Error finding container 811093373bea9fa01ed110d0a5a54dc9aa86c8fb92190a3f34601e32354408ed: Status 404 returned error can't find the container with id 811093373bea9fa01ed110d0a5a54dc9aa86c8fb92190a3f34601e32354408ed
Jul 29 20:07:32.169861 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.169764    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz" event={"ID":"66ffd996-20d7-4e09-84b5-a9e050084100","Type":"ContainerStarted","Data":"2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d"}
Jul 29 20:07:32.170046 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.169930    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:32.171550 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.171524    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89" event={"ID":"6f680b4d-7496-40f5-bb73-d7aeb852e576","Type":"ContainerStarted","Data":"86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d"}
Jul 29 20:07:32.171703 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.171603    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:32.171703 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.171630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89" event={"ID":"6f680b4d-7496-40f5-bb73-d7aeb852e576","Type":"ContainerStarted","Data":"036a40acb135cc519dd251465aeadd444e61336eef322b93183847c322c3b7b3"}
Jul 29 20:07:32.173190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.173165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf" event={"ID":"8a988bfc-5004-4695-9913-a8a269cf2624","Type":"ContainerStarted","Data":"7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487"}
Jul 29 20:07:32.173281 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.173209    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf" event={"ID":"8a988bfc-5004-4695-9913-a8a269cf2624","Type":"ContainerStarted","Data":"811093373bea9fa01ed110d0a5a54dc9aa86c8fb92190a3f34601e32354408ed"}
Jul 29 20:07:32.173361 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.173291    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:32.204963 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.204912    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz" podStartSLOduration=1.515577381 podStartE2EDuration="2.204897241s" totalImagesPullingTime="689.31986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:07:32.187601477 +0000 UTC m=+2258.146096390" watchObservedRunningTime="2026-07-29 20:07:32.204897241 +0000 UTC m=+2258.163392160"
Jul 29 20:07:32.205617 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:32.205560    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89" podStartSLOduration=1.205553752 podStartE2EDuration="1.205553752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:07:32.203661971 +0000 UTC m=+2258.162156887" watchObservedRunningTime="2026-07-29 20:07:32.205553752 +0000 UTC m=+2258.164048646"
Jul 29 20:07:38.184816 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:38.184785    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:07:38.205210 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:38.205160    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf" podStartSLOduration=7.205144964 podStartE2EDuration="7.205144964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:07:32.219810193 +0000 UTC m=+2258.178305111" watchObservedRunningTime="2026-07-29 20:07:38.205144964 +0000 UTC m=+2264.163639857"
Jul 29 20:07:43.180449 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:43.180414    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:07:43.180929 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:43.180730    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:07:58.550537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.550496    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"]
Jul 29 20:07:58.554128 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.554106    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.556160 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.556132    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-620565-serving-cert\""
Jul 29 20:07:58.565084 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.565058    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"]
Jul 29 20:07:58.595431 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.595392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lzl\" (UniqueName: \"kubernetes.io/projected/b686e296-a344-4a19-b56a-b48b41813e2b-kube-api-access-n2lzl\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.595622 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.595454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.595713 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.595632    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b686e296-a344-4a19-b56a-b48b41813e2b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.696998 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.696960    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b686e296-a344-4a19-b56a-b48b41813e2b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.697184 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.697078    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lzl\" (UniqueName: \"kubernetes.io/projected/b686e296-a344-4a19-b56a-b48b41813e2b-kube-api-access-n2lzl\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.697184 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.697106    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.697274 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:07:58.697238    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-serving-cert: secret "maas-api-e2e-inf-b-620565-serving-cert" not found
Jul 29 20:07:58.697324 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:07:58.697316    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls podName:b686e296-a344-4a19-b56a-b48b41813e2b nodeName:}" failed. No retries permitted until 2026-07-29 20:07:59.197295418 +0000 UTC m=+2285.155790295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls") pod "maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" (UID: "b686e296-a344-4a19-b56a-b48b41813e2b") : secret "maas-api-e2e-inf-b-620565-serving-cert" not found
Jul 29 20:07:58.697828 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.697802    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b686e296-a344-4a19-b56a-b48b41813e2b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:58.706316 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:58.706289    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lzl\" (UniqueName: \"kubernetes.io/projected/b686e296-a344-4a19-b56a-b48b41813e2b-kube-api-access-n2lzl\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:59.203488 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.203451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:07:59.203716 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:07:59.203642    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-serving-cert: secret "maas-api-e2e-inf-b-620565-serving-cert" not found
Jul 29 20:07:59.203784 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:07:59.203731    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls podName:b686e296-a344-4a19-b56a-b48b41813e2b nodeName:}" failed. No retries permitted until 2026-07-29 20:08:00.203702518 +0000 UTC m=+2286.162197403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls") pod "maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" (UID: "b686e296-a344-4a19-b56a-b48b41813e2b") : secret "maas-api-e2e-inf-b-620565-serving-cert" not found
Jul 29 20:07:59.425451 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.425414    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"]
Jul 29 20:07:59.428954 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.428937    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.433553 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.433527    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-620565-dockercfg-4hg8s\""
Jul 29 20:07:59.433709 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.433533    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-620565\""
Jul 29 20:07:59.440889 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.440864    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"]
Jul 29 20:07:59.506527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.506437    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75t8v\" (UniqueName: \"kubernetes.io/projected/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-kube-api-access-75t8v\") pod \"payload-processing-e2e-inf-b-620565-799fffb688-q7tp9\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") " pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.506527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.506498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-620565-799fffb688-q7tp9\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") " pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.607423 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.607386    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-620565-799fffb688-q7tp9\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") " pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.607881 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.607529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75t8v\" (UniqueName: \"kubernetes.io/projected/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-kube-api-access-75t8v\") pod \"payload-processing-e2e-inf-b-620565-799fffb688-q7tp9\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") " pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.608183 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.608164    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-620565-799fffb688-q7tp9\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") " pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.618485 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.618455    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75t8v\" (UniqueName: \"kubernetes.io/projected/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-kube-api-access-75t8v\") pod \"payload-processing-e2e-inf-b-620565-799fffb688-q7tp9\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") " pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.741840 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.741801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:07:59.885622 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:07:59.885130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"]
Jul 29 20:07:59.888803 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:07:59.888774    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc806dd35_d49e_49ee_b6b6_08dba2bd9d12.slice/crio-3cc7aecad1d33ba85b531cbdfa8a02b8c0b22e7d88aa9f3ea9d2fcda6110df16 WatchSource:0}: Error finding container 3cc7aecad1d33ba85b531cbdfa8a02b8c0b22e7d88aa9f3ea9d2fcda6110df16: Status 404 returned error can't find the container with id 3cc7aecad1d33ba85b531cbdfa8a02b8c0b22e7d88aa9f3ea9d2fcda6110df16
Jul 29 20:08:00.213725 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.213681    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:08:00.216275 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.216244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") pod \"maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:08:00.308522 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.308473    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9" event={"ID":"c806dd35-d49e-49ee-b6b6-08dba2bd9d12","Type":"ContainerStarted","Data":"10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33"}
Jul 29 20:08:00.308722 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.308542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9" event={"ID":"c806dd35-d49e-49ee-b6b6-08dba2bd9d12","Type":"ContainerStarted","Data":"3cc7aecad1d33ba85b531cbdfa8a02b8c0b22e7d88aa9f3ea9d2fcda6110df16"}
Jul 29 20:08:00.308722 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.308679    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:08:00.324452 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.324374    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9" podStartSLOduration=1.324356682 podStartE2EDuration="1.324356682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:08:00.324114696 +0000 UTC m=+2286.282609611" watchObservedRunningTime="2026-07-29 20:08:00.324356682 +0000 UTC m=+2286.282851579"
Jul 29 20:08:00.367529 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.367500    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:08:00.511522 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:00.511498    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"]
Jul 29 20:08:00.514443 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:08:00.514418    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb686e296_a344_4a19_b56a_b48b41813e2b.slice/crio-9e64dbc2d7122c6dfebf0adb8ae2508a806461203a1ecc752bba652374b0d33c WatchSource:0}: Error finding container 9e64dbc2d7122c6dfebf0adb8ae2508a806461203a1ecc752bba652374b0d33c: Status 404 returned error can't find the container with id 9e64dbc2d7122c6dfebf0adb8ae2508a806461203a1ecc752bba652374b0d33c
Jul 29 20:08:01.314642 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:01.314598    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" event={"ID":"b686e296-a344-4a19-b56a-b48b41813e2b","Type":"ContainerStarted","Data":"761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b"}
Jul 29 20:08:01.315066 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:01.314666    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" event={"ID":"b686e296-a344-4a19-b56a-b48b41813e2b","Type":"ContainerStarted","Data":"9e64dbc2d7122c6dfebf0adb8ae2508a806461203a1ecc752bba652374b0d33c"}
Jul 29 20:08:01.315066 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:01.314826    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:08:01.331280 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:01.331228    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" podStartSLOduration=2.961025855 podStartE2EDuration="3.331213166s" totalImagesPullingTime="370.187311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:08:01.330013036 +0000 UTC m=+2287.288507931" watchObservedRunningTime="2026-07-29 20:08:01.331213166 +0000 UTC m=+2287.289708059"
Jul 29 20:08:07.325019 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:07.324992    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:08:11.317607 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:08:11.317545    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:09:14.673986 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.673948    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"]
Jul 29 20:09:14.677323 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.677300    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.680033 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.679998    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-620565\"/\"openshift-service-ca.crt\""
Jul 29 20:09:14.680367 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.680346    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-620565\"/\"kube-root-ca.crt\""
Jul 29 20:09:14.680477 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.680346    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-620565\"/\"test-model-620565-kserve-self-signed-certs\""
Jul 29 20:09:14.680477 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.680346    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-620565\"/\"default-dockercfg-rtnqj\""
Jul 29 20:09:14.688054 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.688031    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"]
Jul 29 20:09:14.777059 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.777015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-home\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.777059 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.777058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-model-cache\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.777302 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.777078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tmp-dir\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.777302 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.777102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m887s\" (UniqueName: \"kubernetes.io/projected/8f05d7c3-4869-43d6-841e-d89f79fb73ca-kube-api-access-m887s\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.777302 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.777167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-dshm\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.777460 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.777314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tls-certs\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.878739 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.878705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-dshm\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.878933 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.878765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tls-certs\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.878933 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.878825    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-home\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.878933 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.878843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-model-cache\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.879112 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.878971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tmp-dir\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.879112 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.879001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m887s\" (UniqueName: \"kubernetes.io/projected/8f05d7c3-4869-43d6-841e-d89f79fb73ca-kube-api-access-m887s\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.879249 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.879230    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-home\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.879330 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.879309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tmp-dir\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.879386 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.879337    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-model-cache\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.881732 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.881704    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-dshm\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.881924 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.881911    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tls-certs\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.888477 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.888454    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m887s\" (UniqueName: \"kubernetes.io/projected/8f05d7c3-4869-43d6-841e-d89f79fb73ca-kube-api-access-m887s\") pod \"test-model-620565-kserve-99b558557-9m2mb\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") " pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:14.987731 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:14.987699    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:15.127258 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:15.127235    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"]
Jul 29 20:09:15.130053 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:09:15.130023    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f05d7c3_4869_43d6_841e_d89f79fb73ca.slice/crio-7a0016c29def733a0c07e92afad236193f987caf2af75e44b67eaa6dd5a755f6 WatchSource:0}: Error finding container 7a0016c29def733a0c07e92afad236193f987caf2af75e44b67eaa6dd5a755f6: Status 404 returned error can't find the container with id 7a0016c29def733a0c07e92afad236193f987caf2af75e44b67eaa6dd5a755f6
Jul 29 20:09:15.674855 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:15.674809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb" event={"ID":"8f05d7c3-4869-43d6-841e-d89f79fb73ca","Type":"ContainerStarted","Data":"f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966"}
Jul 29 20:09:15.674855 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:15.674858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb" event={"ID":"8f05d7c3-4869-43d6-841e-d89f79fb73ca","Type":"ContainerStarted","Data":"7a0016c29def733a0c07e92afad236193f987caf2af75e44b67eaa6dd5a755f6"}
Jul 29 20:09:15.694288 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:15.694235    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb" podStartSLOduration=1.694217546 podStartE2EDuration="1.694217546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:09:15.691037914 +0000 UTC m=+2361.649532820" watchObservedRunningTime="2026-07-29 20:09:15.694217546 +0000 UTC m=+2361.652712439"
Jul 29 20:09:24.988609 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:24.988543    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:24.989104 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:24.988687    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:25.001749 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:25.001722    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:25.732590 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:25.732548    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:09:54.687752 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:54.687722    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:09:54.691602 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:54.691561    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:09:59.260146 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:09:59.260105    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"]
Jul 29 20:10:09.989803 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:09.989768    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"]
Jul 29 20:10:09.990444 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:09.989989    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz" podUID="66ffd996-20d7-4e09-84b5-a9e050084100" containerName="maas-api" containerID="cri-o://2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d" gracePeriod=30
Jul 29 20:10:10.058519 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.058483    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"]
Jul 29 20:10:10.058853 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.058822    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf" podUID="8a988bfc-5004-4695-9913-a8a269cf2624" containerName="payload-processing" containerID="cri-o://7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487" gracePeriod=30
Jul 29 20:10:10.068472 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.068441    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"]
Jul 29 20:10:10.069410 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.068723    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89" podUID="6f680b4d-7496-40f5-bb73-d7aeb852e576" containerName="payload-pre-processing" containerID="cri-o://86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d" gracePeriod=30
Jul 29 20:10:10.342986 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.342947    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:10:10.366852 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.366821    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:10:10.379185 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.379161    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:10:10.392531 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.392506    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7gsd\" (UniqueName: \"kubernetes.io/projected/8a988bfc-5004-4695-9913-a8a269cf2624-kube-api-access-q7gsd\") pod \"8a988bfc-5004-4695-9913-a8a269cf2624\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") "
Jul 29 20:10:10.392741 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.392627    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99rrc\" (UniqueName: \"kubernetes.io/projected/66ffd996-20d7-4e09-84b5-a9e050084100-kube-api-access-99rrc\") pod \"66ffd996-20d7-4e09-84b5-a9e050084100\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") "
Jul 29 20:10:10.392741 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.392656    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ffd996-20d7-4e09-84b5-a9e050084100-openshift-service-ca\") pod \"66ffd996-20d7-4e09-84b5-a9e050084100\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") "
Jul 29 20:10:10.392741 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.392684    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a988bfc-5004-4695-9913-a8a269cf2624-plugins-config-volume\") pod \"8a988bfc-5004-4695-9913-a8a269cf2624\" (UID: \"8a988bfc-5004-4695-9913-a8a269cf2624\") "
Jul 29 20:10:10.392914 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.392748    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ffd996-20d7-4e09-84b5-a9e050084100-maas-api-tls\") pod \"66ffd996-20d7-4e09-84b5-a9e050084100\" (UID: \"66ffd996-20d7-4e09-84b5-a9e050084100\") "
Jul 29 20:10:10.393062 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.393028    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ffd996-20d7-4e09-84b5-a9e050084100-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66ffd996-20d7-4e09-84b5-a9e050084100" (UID: "66ffd996-20d7-4e09-84b5-a9e050084100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:10:10.393156 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.393131    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a988bfc-5004-4695-9913-a8a269cf2624-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a988bfc-5004-4695-9913-a8a269cf2624" (UID: "8a988bfc-5004-4695-9913-a8a269cf2624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:10:10.394928 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.394900    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ffd996-20d7-4e09-84b5-a9e050084100-kube-api-access-99rrc" (OuterVolumeSpecName: "kube-api-access-99rrc") pod "66ffd996-20d7-4e09-84b5-a9e050084100" (UID: "66ffd996-20d7-4e09-84b5-a9e050084100"). InnerVolumeSpecName "kube-api-access-99rrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:10:10.395211 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.395172    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ffd996-20d7-4e09-84b5-a9e050084100-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66ffd996-20d7-4e09-84b5-a9e050084100" (UID: "66ffd996-20d7-4e09-84b5-a9e050084100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:10:10.395302 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.395263    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a988bfc-5004-4695-9913-a8a269cf2624-kube-api-access-q7gsd" (OuterVolumeSpecName: "kube-api-access-q7gsd") pod "8a988bfc-5004-4695-9913-a8a269cf2624" (UID: "8a988bfc-5004-4695-9913-a8a269cf2624"). InnerVolumeSpecName "kube-api-access-q7gsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:10:10.493951 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.493855    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f680b4d-7496-40f5-bb73-d7aeb852e576-plugins-config-volume\") pod \"6f680b4d-7496-40f5-bb73-d7aeb852e576\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") "
Jul 29 20:10:10.493951 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.493906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkp6j\" (UniqueName: \"kubernetes.io/projected/6f680b4d-7496-40f5-bb73-d7aeb852e576-kube-api-access-gkp6j\") pod \"6f680b4d-7496-40f5-bb73-d7aeb852e576\" (UID: \"6f680b4d-7496-40f5-bb73-d7aeb852e576\") "
Jul 29 20:10:10.494190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.494117    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99rrc\" (UniqueName: \"kubernetes.io/projected/66ffd996-20d7-4e09-84b5-a9e050084100-kube-api-access-99rrc\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.494190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.494129    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ffd996-20d7-4e09-84b5-a9e050084100-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.494190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.494138    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a988bfc-5004-4695-9913-a8a269cf2624-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.494190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.494146    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ffd996-20d7-4e09-84b5-a9e050084100-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.494190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.494155    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gsd\" (UniqueName: \"kubernetes.io/projected/8a988bfc-5004-4695-9913-a8a269cf2624-kube-api-access-q7gsd\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.494409 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.494192    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f680b4d-7496-40f5-bb73-d7aeb852e576-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f680b4d-7496-40f5-bb73-d7aeb852e576" (UID: "6f680b4d-7496-40f5-bb73-d7aeb852e576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:10:10.496260 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.496232    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f680b4d-7496-40f5-bb73-d7aeb852e576-kube-api-access-gkp6j" (OuterVolumeSpecName: "kube-api-access-gkp6j") pod "6f680b4d-7496-40f5-bb73-d7aeb852e576" (UID: "6f680b4d-7496-40f5-bb73-d7aeb852e576"). InnerVolumeSpecName "kube-api-access-gkp6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:10:10.594941 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.594893    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f680b4d-7496-40f5-bb73-d7aeb852e576-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.594941 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.594934    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkp6j\" (UniqueName: \"kubernetes.io/projected/6f680b4d-7496-40f5-bb73-d7aeb852e576-kube-api-access-gkp6j\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:10.944841 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.944806    2555 generic.go:356] "Generic (PLEG): container finished" podID="66ffd996-20d7-4e09-84b5-a9e050084100" containerID="2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d" exitCode=0
Jul 29 20:10:10.945024 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.944856    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"
Jul 29 20:10:10.945024 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.944894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz" event={"ID":"66ffd996-20d7-4e09-84b5-a9e050084100","Type":"ContainerDied","Data":"2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d"}
Jul 29 20:10:10.945024 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.944951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz" event={"ID":"66ffd996-20d7-4e09-84b5-a9e050084100","Type":"ContainerDied","Data":"e165406bf9572b990dc51df8d3b8bc44a34c6ab50c408653cd21097376be69ed"}
Jul 29 20:10:10.945024 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.944980    2555 scope.go:117] "RemoveContainer" containerID="2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d"
Jul 29 20:10:10.946357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.946334    2555 generic.go:356] "Generic (PLEG): container finished" podID="6f680b4d-7496-40f5-bb73-d7aeb852e576" containerID="86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d" exitCode=0
Jul 29 20:10:10.946463 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.946379    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"
Jul 29 20:10:10.946463 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.946398    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89" event={"ID":"6f680b4d-7496-40f5-bb73-d7aeb852e576","Type":"ContainerDied","Data":"86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d"}
Jul 29 20:10:10.946463 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.946436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89" event={"ID":"6f680b4d-7496-40f5-bb73-d7aeb852e576","Type":"ContainerDied","Data":"036a40acb135cc519dd251465aeadd444e61336eef322b93183847c322c3b7b3"}
Jul 29 20:10:10.948254 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.948230    2555 generic.go:356] "Generic (PLEG): container finished" podID="8a988bfc-5004-4695-9913-a8a269cf2624" containerID="7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487" exitCode=0
Jul 29 20:10:10.948396 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.948257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf" event={"ID":"8a988bfc-5004-4695-9913-a8a269cf2624","Type":"ContainerDied","Data":"7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487"}
Jul 29 20:10:10.948396 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.948271    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"
Jul 29 20:10:10.948396 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.948305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf" event={"ID":"8a988bfc-5004-4695-9913-a8a269cf2624","Type":"ContainerDied","Data":"811093373bea9fa01ed110d0a5a54dc9aa86c8fb92190a3f34601e32354408ed"}
Jul 29 20:10:10.955053 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.955034    2555 scope.go:117] "RemoveContainer" containerID="2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d"
Jul 29 20:10:10.955349 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:10.955331    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d\": container with ID starting with 2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d not found: ID does not exist" containerID="2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d"
Jul 29 20:10:10.955408 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.955356    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d"} err="failed to get container status \"2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d\": rpc error: code = NotFound desc = could not find container \"2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d\": container with ID starting with 2f8db40bfe54519146511591a8e50623ef4f978920964e9567cf35534b232e1d not found: ID does not exist"
Jul 29 20:10:10.955408 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.955375    2555 scope.go:117] "RemoveContainer" containerID="86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d"
Jul 29 20:10:10.964060 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.964041    2555 scope.go:117] "RemoveContainer" containerID="86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d"
Jul 29 20:10:10.964347 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:10.964326    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d\": container with ID starting with 86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d not found: ID does not exist" containerID="86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d"
Jul 29 20:10:10.964418 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.964358    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d"} err="failed to get container status \"86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d\": rpc error: code = NotFound desc = could not find container \"86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d\": container with ID starting with 86fd01b5287a8bf82a8d2f5e0ec9eb3703a62888991cc23393dc874fd6f8514d not found: ID does not exist"
Jul 29 20:10:10.964418 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.964383    2555 scope.go:117] "RemoveContainer" containerID="7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487"
Jul 29 20:10:10.971865 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.971838    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"]
Jul 29 20:10:10.972323 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.972310    2555 scope.go:117] "RemoveContainer" containerID="7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487"
Jul 29 20:10:10.972605 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:10.972569    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487\": container with ID starting with 7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487 not found: ID does not exist" containerID="7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487"
Jul 29 20:10:10.972703 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.972613    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487"} err="failed to get container status \"7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487\": rpc error: code = NotFound desc = could not find container \"7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487\": container with ID starting with 7e80f4c642595fd2ef4b09ba23de4f3844067aec97eb5e184aa556a05e94b487 not found: ID does not exist"
Jul 29 20:10:10.977306 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.977285    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f5ad4c-6747c9d75b-pxh89"]
Jul 29 20:10:10.990736 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.990712    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"]
Jul 29 20:10:10.998546 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:10.998525    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f5ad4c-75cf7dbc8d-vnmlz"]
Jul 29 20:10:11.011979 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:11.011957    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"]
Jul 29 20:10:11.017121 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:11.017100    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f5ad4c-5599d56784-gbpvf"]
Jul 29 20:10:12.658922 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:12.658888    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ffd996-20d7-4e09-84b5-a9e050084100" path="/var/lib/kubelet/pods/66ffd996-20d7-4e09-84b5-a9e050084100/volumes"
Jul 29 20:10:12.659286 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:12.659249    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f680b4d-7496-40f5-bb73-d7aeb852e576" path="/var/lib/kubelet/pods/6f680b4d-7496-40f5-bb73-d7aeb852e576/volumes"
Jul 29 20:10:12.659630 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:12.659615    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a988bfc-5004-4695-9913-a8a269cf2624" path="/var/lib/kubelet/pods/8a988bfc-5004-4695-9913-a8a269cf2624/volumes"
Jul 29 20:10:14.279786 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.279742    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb" podUID="8f05d7c3-4869-43d6-841e-d89f79fb73ca" containerName="main" containerID="cri-o://f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966" gracePeriod=45
Jul 29 20:10:14.533204 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.533143    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:10:14.631986 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.631952    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-dshm\") pod \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") "
Jul 29 20:10:14.632150 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632025    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-model-cache\") pod \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") "
Jul 29 20:10:14.632150 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632045    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tls-certs\") pod \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") "
Jul 29 20:10:14.632150 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632077    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-home\") pod \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") "
Jul 29 20:10:14.632150 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632123    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tmp-dir\") pod \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") "
Jul 29 20:10:14.632317 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632154    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m887s\" (UniqueName: \"kubernetes.io/projected/8f05d7c3-4869-43d6-841e-d89f79fb73ca-kube-api-access-m887s\") pod \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\" (UID: \"8f05d7c3-4869-43d6-841e-d89f79fb73ca\") "
Jul 29 20:10:14.632375 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632307    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-home" (OuterVolumeSpecName: "home") pod "8f05d7c3-4869-43d6-841e-d89f79fb73ca" (UID: "8f05d7c3-4869-43d6-841e-d89f79fb73ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:10:14.632375 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632321    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f05d7c3-4869-43d6-841e-d89f79fb73ca" (UID: "8f05d7c3-4869-43d6-841e-d89f79fb73ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:10:14.632485 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632382    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f05d7c3-4869-43d6-841e-d89f79fb73ca" (UID: "8f05d7c3-4869-43d6-841e-d89f79fb73ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:10:14.632567 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632548    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:14.632656 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632571    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:14.632656 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.632610    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:14.634437 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.634414    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-dshm" (OuterVolumeSpecName: "dshm") pod "8f05d7c3-4869-43d6-841e-d89f79fb73ca" (UID: "8f05d7c3-4869-43d6-841e-d89f79fb73ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:10:14.634520 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.634449    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f05d7c3-4869-43d6-841e-d89f79fb73ca" (UID: "8f05d7c3-4869-43d6-841e-d89f79fb73ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:10:14.634567 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.634547    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f05d7c3-4869-43d6-841e-d89f79fb73ca-kube-api-access-m887s" (OuterVolumeSpecName: "kube-api-access-m887s") pod "8f05d7c3-4869-43d6-841e-d89f79fb73ca" (UID: "8f05d7c3-4869-43d6-841e-d89f79fb73ca"). InnerVolumeSpecName "kube-api-access-m887s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:10:14.733057 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.733013    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f05d7c3-4869-43d6-841e-d89f79fb73ca-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:14.733057 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.733053    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m887s\" (UniqueName: \"kubernetes.io/projected/8f05d7c3-4869-43d6-841e-d89f79fb73ca-kube-api-access-m887s\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:14.733057 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.733064    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f05d7c3-4869-43d6-841e-d89f79fb73ca-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:14.971045 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.971011    2555 generic.go:356] "Generic (PLEG): container finished" podID="8f05d7c3-4869-43d6-841e-d89f79fb73ca" containerID="f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966" exitCode=0
Jul 29 20:10:14.971248 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.971071    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"
Jul 29 20:10:14.971248 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.971076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb" event={"ID":"8f05d7c3-4869-43d6-841e-d89f79fb73ca","Type":"ContainerDied","Data":"f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966"}
Jul 29 20:10:14.971248 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.971116    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb" event={"ID":"8f05d7c3-4869-43d6-841e-d89f79fb73ca","Type":"ContainerDied","Data":"7a0016c29def733a0c07e92afad236193f987caf2af75e44b67eaa6dd5a755f6"}
Jul 29 20:10:14.971248 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.971140    2555 scope.go:117] "RemoveContainer" containerID="f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966"
Jul 29 20:10:14.981818 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.981764    2555 scope.go:117] "RemoveContainer" containerID="f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966"
Jul 29 20:10:14.982269 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:14.982247    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966\": container with ID starting with f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966 not found: ID does not exist" containerID="f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966"
Jul 29 20:10:14.982350 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.982276    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966"} err="failed to get container status \"f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966\": rpc error: code = NotFound desc = could not find container \"f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966\": container with ID starting with f29e7849d2edd55fb6d764a978ec517c53f63ba874aa93bab21e1d15a943b966 not found: ID does not exist"
Jul 29 20:10:14.988974 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.988948    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"]
Jul 29 20:10:14.994366 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:14.994344    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-620565/test-model-620565-kserve-99b558557-9m2mb"]
Jul 29 20:10:16.659099 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:16.659064    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f05d7c3-4869-43d6-841e-d89f79fb73ca" path="/var/lib/kubelet/pods/8f05d7c3-4869-43d6-841e-d89f79fb73ca/volumes"
Jul 29 20:10:28.186744 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.186711    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"]
Jul 29 20:10:28.187199 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.186931    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" podUID="b686e296-a344-4a19-b56a-b48b41813e2b" containerName="maas-api" containerID="cri-o://761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b" gracePeriod=30
Jul 29 20:10:28.248705 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.248667    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"]
Jul 29 20:10:28.248967 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.248934    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9" podUID="c806dd35-d49e-49ee-b6b6-08dba2bd9d12" containerName="payload-processing" containerID="cri-o://10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33" gracePeriod=30
Jul 29 20:10:28.501485 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.501452    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:10:28.554034 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.554009    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:10:28.665977 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.665946    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b686e296-a344-4a19-b56a-b48b41813e2b-openshift-service-ca\") pod \"b686e296-a344-4a19-b56a-b48b41813e2b\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") "
Jul 29 20:10:28.666125 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666047    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") pod \"b686e296-a344-4a19-b56a-b48b41813e2b\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") "
Jul 29 20:10:28.666125 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666086    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2lzl\" (UniqueName: \"kubernetes.io/projected/b686e296-a344-4a19-b56a-b48b41813e2b-kube-api-access-n2lzl\") pod \"b686e296-a344-4a19-b56a-b48b41813e2b\" (UID: \"b686e296-a344-4a19-b56a-b48b41813e2b\") "
Jul 29 20:10:28.666212 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-plugins-config-volume\") pod \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") "
Jul 29 20:10:28.666212 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666171    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75t8v\" (UniqueName: \"kubernetes.io/projected/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-kube-api-access-75t8v\") pod \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\" (UID: \"c806dd35-d49e-49ee-b6b6-08dba2bd9d12\") "
Jul 29 20:10:28.666325 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666298    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b686e296-a344-4a19-b56a-b48b41813e2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b686e296-a344-4a19-b56a-b48b41813e2b" (UID: "b686e296-a344-4a19-b56a-b48b41813e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:10:28.666537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666515    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b686e296-a344-4a19-b56a-b48b41813e2b-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:28.666676 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.666548    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c806dd35-d49e-49ee-b6b6-08dba2bd9d12" (UID: "c806dd35-d49e-49ee-b6b6-08dba2bd9d12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:10:28.668359 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.668335    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b686e296-a344-4a19-b56a-b48b41813e2b-kube-api-access-n2lzl" (OuterVolumeSpecName: "kube-api-access-n2lzl") pod "b686e296-a344-4a19-b56a-b48b41813e2b" (UID: "b686e296-a344-4a19-b56a-b48b41813e2b"). InnerVolumeSpecName "kube-api-access-n2lzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:10:28.668359 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.668351    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b686e296-a344-4a19-b56a-b48b41813e2b" (UID: "b686e296-a344-4a19-b56a-b48b41813e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:10:28.668470 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.668362    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-kube-api-access-75t8v" (OuterVolumeSpecName: "kube-api-access-75t8v") pod "c806dd35-d49e-49ee-b6b6-08dba2bd9d12" (UID: "c806dd35-d49e-49ee-b6b6-08dba2bd9d12"). InnerVolumeSpecName "kube-api-access-75t8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:10:28.767670 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.767571    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2lzl\" (UniqueName: \"kubernetes.io/projected/b686e296-a344-4a19-b56a-b48b41813e2b-kube-api-access-n2lzl\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:28.767670 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.767627    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:28.767670 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.767639    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75t8v\" (UniqueName: \"kubernetes.io/projected/c806dd35-d49e-49ee-b6b6-08dba2bd9d12-kube-api-access-75t8v\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:28.767670 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:28.767648    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b686e296-a344-4a19-b56a-b48b41813e2b-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:10:29.039762 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.039669    2555 generic.go:356] "Generic (PLEG): container finished" podID="b686e296-a344-4a19-b56a-b48b41813e2b" containerID="761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b" exitCode=0
Jul 29 20:10:29.039762 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.039717    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"
Jul 29 20:10:29.039991 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.039755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" event={"ID":"b686e296-a344-4a19-b56a-b48b41813e2b","Type":"ContainerDied","Data":"761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b"}
Jul 29 20:10:29.039991 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.039814    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9" event={"ID":"b686e296-a344-4a19-b56a-b48b41813e2b","Type":"ContainerDied","Data":"9e64dbc2d7122c6dfebf0adb8ae2508a806461203a1ecc752bba652374b0d33c"}
Jul 29 20:10:29.039991 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.039845    2555 scope.go:117] "RemoveContainer" containerID="761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b"
Jul 29 20:10:29.041230 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.041208    2555 generic.go:356] "Generic (PLEG): container finished" podID="c806dd35-d49e-49ee-b6b6-08dba2bd9d12" containerID="10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33" exitCode=0
Jul 29 20:10:29.041334 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.041257    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"
Jul 29 20:10:29.041334 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.041294    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9" event={"ID":"c806dd35-d49e-49ee-b6b6-08dba2bd9d12","Type":"ContainerDied","Data":"10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33"}
Jul 29 20:10:29.041433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.041336    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9" event={"ID":"c806dd35-d49e-49ee-b6b6-08dba2bd9d12","Type":"ContainerDied","Data":"3cc7aecad1d33ba85b531cbdfa8a02b8c0b22e7d88aa9f3ea9d2fcda6110df16"}
Jul 29 20:10:29.049113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.048981    2555 scope.go:117] "RemoveContainer" containerID="761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b"
Jul 29 20:10:29.049611 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:29.049563    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b\": container with ID starting with 761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b not found: ID does not exist" containerID="761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b"
Jul 29 20:10:29.049701 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.049621    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b"} err="failed to get container status \"761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b\": rpc error: code = NotFound desc = could not find container \"761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b\": container with ID starting with 761021a353a2e928a581cc1a4c6d1fe010ec74b35aa2fcba9c646a20420cc97b not found: ID does not exist"
Jul 29 20:10:29.049701 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.049639    2555 scope.go:117] "RemoveContainer" containerID="10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33"
Jul 29 20:10:29.057990 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.057976    2555 scope.go:117] "RemoveContainer" containerID="10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33"
Jul 29 20:10:29.058262 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:29.058238    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33\": container with ID starting with 10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33 not found: ID does not exist" containerID="10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33"
Jul 29 20:10:29.058359 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.058270    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33"} err="failed to get container status \"10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33\": rpc error: code = NotFound desc = could not find container \"10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33\": container with ID starting with 10272d714626974f3e85597926e0b67ba16b6036418dca0fbf5a323b8767ab33 not found: ID does not exist"
Jul 29 20:10:29.071866 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.071734    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"]
Jul 29 20:10:29.073814 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.073793    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-620565-6fcfdc9c6-zzgf9"]
Jul 29 20:10:29.091340 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.091308    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"]
Jul 29 20:10:29.095391 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:29.095371    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-620565-799fffb688-q7tp9"]
Jul 29 20:10:30.658341 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:30.658305    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b686e296-a344-4a19-b56a-b48b41813e2b" path="/var/lib/kubelet/pods/b686e296-a344-4a19-b56a-b48b41813e2b/volumes"
Jul 29 20:10:30.659048 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:30.659021    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c806dd35-d49e-49ee-b6b6-08dba2bd9d12" path="/var/lib/kubelet/pods/c806dd35-d49e-49ee-b6b6-08dba2bd9d12/volumes"
Jul 29 20:10:39.442808 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.442771    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"]
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443130    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c806dd35-d49e-49ee-b6b6-08dba2bd9d12" containerName="payload-processing"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443143    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c806dd35-d49e-49ee-b6b6-08dba2bd9d12" containerName="payload-processing"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443159    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b686e296-a344-4a19-b56a-b48b41813e2b" containerName="maas-api"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443169    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b686e296-a344-4a19-b56a-b48b41813e2b" containerName="maas-api"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443181    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f05d7c3-4869-43d6-841e-d89f79fb73ca" containerName="main"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443198    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f05d7c3-4869-43d6-841e-d89f79fb73ca" containerName="main"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443214    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f680b4d-7496-40f5-bb73-d7aeb852e576" containerName="payload-pre-processing"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443221    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f680b4d-7496-40f5-bb73-d7aeb852e576" containerName="payload-pre-processing"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443228    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a988bfc-5004-4695-9913-a8a269cf2624" containerName="payload-processing"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443233    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a988bfc-5004-4695-9913-a8a269cf2624" containerName="payload-processing"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443244    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ffd996-20d7-4e09-84b5-a9e050084100" containerName="maas-api"
Jul 29 20:10:39.443289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443249    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ffd996-20d7-4e09-84b5-a9e050084100" containerName="maas-api"
Jul 29 20:10:39.443901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443308    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c806dd35-d49e-49ee-b6b6-08dba2bd9d12" containerName="payload-processing"
Jul 29 20:10:39.443901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443316    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b686e296-a344-4a19-b56a-b48b41813e2b" containerName="maas-api"
Jul 29 20:10:39.443901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443320    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ffd996-20d7-4e09-84b5-a9e050084100" containerName="maas-api"
Jul 29 20:10:39.443901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443326    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a988bfc-5004-4695-9913-a8a269cf2624" containerName="payload-processing"
Jul 29 20:10:39.443901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443334    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f680b4d-7496-40f5-bb73-d7aeb852e576" containerName="payload-pre-processing"
Jul 29 20:10:39.443901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.443338    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f05d7c3-4869-43d6-841e-d89f79fb73ca" containerName="main"
Jul 29 20:10:39.447157 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.447125    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.449120 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.449096    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-78b35e-serving-cert\""
Jul 29 20:10:39.449238 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.449217    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 20:10:39.453922 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.453890    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"]
Jul 29 20:10:39.569595 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.569549    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95181b1e-95b2-4358-bfb7-7257e9baa6d9-openshift-service-ca\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.569770 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.569703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.569820 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.569791    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9kq\" (UniqueName: \"kubernetes.io/projected/95181b1e-95b2-4358-bfb7-7257e9baa6d9-kube-api-access-sq9kq\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.671122 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.671068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9kq\" (UniqueName: \"kubernetes.io/projected/95181b1e-95b2-4358-bfb7-7257e9baa6d9-kube-api-access-sq9kq\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.671325 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.671136    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95181b1e-95b2-4358-bfb7-7257e9baa6d9-openshift-service-ca\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.671325 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.671182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.671325 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:39.671293    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-serving-cert: secret "maas-api-e2e-api-a-78b35e-serving-cert" not found
Jul 29 20:10:39.671506 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:10:39.671360    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls podName:95181b1e-95b2-4358-bfb7-7257e9baa6d9 nodeName:}" failed. No retries permitted until 2026-07-29 20:10:40.171342965 +0000 UTC m=+2446.129837855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls") pod "maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" (UID: "95181b1e-95b2-4358-bfb7-7257e9baa6d9") : secret "maas-api-e2e-api-a-78b35e-serving-cert" not found
Jul 29 20:10:39.671996 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.671968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95181b1e-95b2-4358-bfb7-7257e9baa6d9-openshift-service-ca\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:39.682603 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:39.682561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9kq\" (UniqueName: \"kubernetes.io/projected/95181b1e-95b2-4358-bfb7-7257e9baa6d9-kube-api-access-sq9kq\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:40.175968 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.175930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:40.178523 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.178485    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls\") pod \"maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:40.253964 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.253931    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"]
Jul 29 20:10:40.257285 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.257268    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.259157 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.259128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-78b35e\""
Jul 29 20:10:40.259313 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.259291    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-78b35e-dockercfg-mj4sn\""
Jul 29 20:10:40.267902 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.267876    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"]
Jul 29 20:10:40.362214 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.362179    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:40.377967 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.377935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxh9r\" (UniqueName: \"kubernetes.io/projected/3045d486-0a19-40ed-87b4-de63ffc1e9d3-kube-api-access-qxh9r\") pod \"payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") " pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.378114 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.378015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3045d486-0a19-40ed-87b4-de63ffc1e9d3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") " pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.478608 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.478539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3045d486-0a19-40ed-87b4-de63ffc1e9d3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") " pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.479048 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.478719    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxh9r\" (UniqueName: \"kubernetes.io/projected/3045d486-0a19-40ed-87b4-de63ffc1e9d3-kube-api-access-qxh9r\") pod \"payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") " pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.479855 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.479836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3045d486-0a19-40ed-87b4-de63ffc1e9d3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") " pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.488626 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.488595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxh9r\" (UniqueName: \"kubernetes.io/projected/3045d486-0a19-40ed-87b4-de63ffc1e9d3-kube-api-access-qxh9r\") pod \"payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") " pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.508840 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.508813    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"]
Jul 29 20:10:40.511434 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:10:40.511398    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95181b1e_95b2_4358_bfb7_7257e9baa6d9.slice/crio-032d98a095738a403771a9db634795ce2b201dad5bb7b5466ddd30462edb6dbd WatchSource:0}: Error finding container 032d98a095738a403771a9db634795ce2b201dad5bb7b5466ddd30462edb6dbd: Status 404 returned error can't find the container with id 032d98a095738a403771a9db634795ce2b201dad5bb7b5466ddd30462edb6dbd
Jul 29 20:10:40.512793 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.512773    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:10:40.568991 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.568957    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:40.713106 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:40.713082    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"]
Jul 29 20:10:40.715026 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:10:40.714992    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3045d486_0a19_40ed_87b4_de63ffc1e9d3.slice/crio-df23fe27a66eb6236e3e1d31befff158368b6ce9fdd82400640f79c670b34c62 WatchSource:0}: Error finding container df23fe27a66eb6236e3e1d31befff158368b6ce9fdd82400640f79c670b34c62: Status 404 returned error can't find the container with id df23fe27a66eb6236e3e1d31befff158368b6ce9fdd82400640f79c670b34c62
Jul 29 20:10:41.102131 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:41.102024    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" event={"ID":"95181b1e-95b2-4358-bfb7-7257e9baa6d9","Type":"ContainerStarted","Data":"032d98a095738a403771a9db634795ce2b201dad5bb7b5466ddd30462edb6dbd"}
Jul 29 20:10:41.103547 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:41.103519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf" event={"ID":"3045d486-0a19-40ed-87b4-de63ffc1e9d3","Type":"ContainerStarted","Data":"7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21"}
Jul 29 20:10:41.103717 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:41.103572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf" event={"ID":"3045d486-0a19-40ed-87b4-de63ffc1e9d3","Type":"ContainerStarted","Data":"df23fe27a66eb6236e3e1d31befff158368b6ce9fdd82400640f79c670b34c62"}
Jul 29 20:10:41.103717 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:41.103662    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:10:41.119590 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:41.119543    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf" podStartSLOduration=1.119528752 podStartE2EDuration="1.119528752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:10:41.118810879 +0000 UTC m=+2447.077305769" watchObservedRunningTime="2026-07-29 20:10:41.119528752 +0000 UTC m=+2447.078023646"
Jul 29 20:10:42.110664 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:42.110626    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" event={"ID":"95181b1e-95b2-4358-bfb7-7257e9baa6d9","Type":"ContainerStarted","Data":"f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2"}
Jul 29 20:10:42.111090 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:42.110712    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:42.127772 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:42.127725    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" podStartSLOduration=2.423204101 podStartE2EDuration="3.127710825s" totalImagesPullingTime="704.506724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:10:42.126857783 +0000 UTC m=+2448.085352677" watchObservedRunningTime="2026-07-29 20:10:42.127710825 +0000 UTC m=+2448.086205721"
Jul 29 20:10:48.121713 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:48.121683    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:10:52.112948 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:10:52.112914    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:11:08.507137 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.507101    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"]
Jul 29 20:11:08.511013 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.510986    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.512902 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.512877    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d3ceb2-serving-cert\""
Jul 29 20:11:08.524836 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.524809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"]
Jul 29 20:11:08.628282 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.628244    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06613625-c97c-4440-a3ff-c268dd3a3ba1-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.628458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.628297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfv8\" (UniqueName: \"kubernetes.io/projected/06613625-c97c-4440-a3ff-c268dd3a3ba1-kube-api-access-wzfv8\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.628458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.628329    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.729643 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.729571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06613625-c97c-4440-a3ff-c268dd3a3ba1-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.729815 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.729673    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfv8\" (UniqueName: \"kubernetes.io/projected/06613625-c97c-4440-a3ff-c268dd3a3ba1-kube-api-access-wzfv8\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.729815 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.729704    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.729909 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:11:08.729868    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-serving-cert: secret "maas-api-e2e-api-b-d3ceb2-serving-cert" not found
Jul 29 20:11:08.729961 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:11:08.729916    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls podName:06613625-c97c-4440-a3ff-c268dd3a3ba1 nodeName:}" failed. No retries permitted until 2026-07-29 20:11:09.229901563 +0000 UTC m=+2475.188396434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls") pod "maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" (UID: "06613625-c97c-4440-a3ff-c268dd3a3ba1") : secret "maas-api-e2e-api-b-d3ceb2-serving-cert" not found
Jul 29 20:11:08.730344 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.730321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06613625-c97c-4440-a3ff-c268dd3a3ba1-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:08.741084 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:08.741050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfv8\" (UniqueName: \"kubernetes.io/projected/06613625-c97c-4440-a3ff-c268dd3a3ba1-kube-api-access-wzfv8\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:09.234665 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.234632    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:09.237424 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.237395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls\") pod \"maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:09.347034 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.346997    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"]
Jul 29 20:11:09.350829 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.350810    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.352858 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.352836    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d3ceb2\""
Jul 29 20:11:09.353019 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.352923    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d3ceb2-dockercfg-fbjlk\""
Jul 29 20:11:09.360348 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.360325    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"]
Jul 29 20:11:09.423201 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.423169    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:09.436904 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.436878    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmwr\" (UniqueName: \"kubernetes.io/projected/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-kube-api-access-5qmwr\") pod \"payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.436904 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.436912    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.537820 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.537789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmwr\" (UniqueName: \"kubernetes.io/projected/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-kube-api-access-5qmwr\") pod \"payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.537820 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.537830    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.538858 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.538824    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.546691 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.546664    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmwr\" (UniqueName: \"kubernetes.io/projected/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-kube-api-access-5qmwr\") pod \"payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.573559 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.573532    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"]
Jul 29 20:11:09.575800 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:11:09.575770    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06613625_c97c_4440_a3ff_c268dd3a3ba1.slice/crio-2604df97f803ab36ff749e5392a0ceaa69228d1276372dd59674b071666b8c20 WatchSource:0}: Error finding container 2604df97f803ab36ff749e5392a0ceaa69228d1276372dd59674b071666b8c20: Status 404 returned error can't find the container with id 2604df97f803ab36ff749e5392a0ceaa69228d1276372dd59674b071666b8c20
Jul 29 20:11:09.664212 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.664174    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:09.810959 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:09.810930    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"]
Jul 29 20:11:09.813404 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:11:09.813377    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d46a95_2dd6_4ff3_a475_1fbf07e98bd3.slice/crio-e2d2b64c96fd87729e8d5af08dfb9f8be97695536e33982c9cb55bcf0fa61f26 WatchSource:0}: Error finding container e2d2b64c96fd87729e8d5af08dfb9f8be97695536e33982c9cb55bcf0fa61f26: Status 404 returned error can't find the container with id e2d2b64c96fd87729e8d5af08dfb9f8be97695536e33982c9cb55bcf0fa61f26
Jul 29 20:11:10.259542 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.259509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" event={"ID":"06613625-c97c-4440-a3ff-c268dd3a3ba1","Type":"ContainerStarted","Data":"7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783"}
Jul 29 20:11:10.259749 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.259563    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" event={"ID":"06613625-c97c-4440-a3ff-c268dd3a3ba1","Type":"ContainerStarted","Data":"2604df97f803ab36ff749e5392a0ceaa69228d1276372dd59674b071666b8c20"}
Jul 29 20:11:10.259749 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.259699    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:10.261213 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.261186    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg" event={"ID":"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3","Type":"ContainerStarted","Data":"508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58"}
Jul 29 20:11:10.261323 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.261237    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg" event={"ID":"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3","Type":"ContainerStarted","Data":"e2d2b64c96fd87729e8d5af08dfb9f8be97695536e33982c9cb55bcf0fa61f26"}
Jul 29 20:11:10.261323 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.261265    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:10.277129 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.277081    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" podStartSLOduration=1.8997373130000001 podStartE2EDuration="2.277066671s" totalImagesPullingTime="377.329358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:11:10.27625104 +0000 UTC m=+2476.234745933" watchObservedRunningTime="2026-07-29 20:11:10.277066671 +0000 UTC m=+2476.235561565"
Jul 29 20:11:10.293534 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:10.293490    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg" podStartSLOduration=1.293474797 podStartE2EDuration="1.293474797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:11:10.291931775 +0000 UTC m=+2476.250426694" watchObservedRunningTime="2026-07-29 20:11:10.293474797 +0000 UTC m=+2476.251969691"
Jul 29 20:11:16.270923 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:16.270886    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:11:21.267977 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:21.267939    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:11:45.875391 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:45.875247    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"]
Jul 29 20:11:45.875970 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:45.875519    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" podUID="95181b1e-95b2-4358-bfb7-7257e9baa6d9" containerName="maas-api" containerID="cri-o://f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2" gracePeriod=30
Jul 29 20:11:45.935889 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:45.935840    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"]
Jul 29 20:11:45.936107 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:45.936080    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf" podUID="3045d486-0a19-40ed-87b4-de63ffc1e9d3" containerName="payload-processing" containerID="cri-o://7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21" gracePeriod=30
Jul 29 20:11:46.231931 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.231904    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:11:46.236219 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.236197    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:11:46.371325 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371265    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3045d486-0a19-40ed-87b4-de63ffc1e9d3-plugins-config-volume\") pod \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") "
Jul 29 20:11:46.371537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371341    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls\") pod \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") "
Jul 29 20:11:46.371537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371412    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95181b1e-95b2-4358-bfb7-7257e9baa6d9-openshift-service-ca\") pod \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") "
Jul 29 20:11:46.371537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371454    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxh9r\" (UniqueName: \"kubernetes.io/projected/3045d486-0a19-40ed-87b4-de63ffc1e9d3-kube-api-access-qxh9r\") pod \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\" (UID: \"3045d486-0a19-40ed-87b4-de63ffc1e9d3\") "
Jul 29 20:11:46.371537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371517    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq9kq\" (UniqueName: \"kubernetes.io/projected/95181b1e-95b2-4358-bfb7-7257e9baa6d9-kube-api-access-sq9kq\") pod \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\" (UID: \"95181b1e-95b2-4358-bfb7-7257e9baa6d9\") "
Jul 29 20:11:46.371892 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371859    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3045d486-0a19-40ed-87b4-de63ffc1e9d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3045d486-0a19-40ed-87b4-de63ffc1e9d3" (UID: "3045d486-0a19-40ed-87b4-de63ffc1e9d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:11:46.371892 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.371901    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95181b1e-95b2-4358-bfb7-7257e9baa6d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95181b1e-95b2-4358-bfb7-7257e9baa6d9" (UID: "95181b1e-95b2-4358-bfb7-7257e9baa6d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:11:46.372276 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.372255    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3045d486-0a19-40ed-87b4-de63ffc1e9d3-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:11:46.372276 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.372276    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95181b1e-95b2-4358-bfb7-7257e9baa6d9-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:11:46.373778 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.373750    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3045d486-0a19-40ed-87b4-de63ffc1e9d3-kube-api-access-qxh9r" (OuterVolumeSpecName: "kube-api-access-qxh9r") pod "3045d486-0a19-40ed-87b4-de63ffc1e9d3" (UID: "3045d486-0a19-40ed-87b4-de63ffc1e9d3"). InnerVolumeSpecName "kube-api-access-qxh9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:11:46.373778 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.373774    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95181b1e-95b2-4358-bfb7-7257e9baa6d9" (UID: "95181b1e-95b2-4358-bfb7-7257e9baa6d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:11:46.374059 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.374033    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95181b1e-95b2-4358-bfb7-7257e9baa6d9-kube-api-access-sq9kq" (OuterVolumeSpecName: "kube-api-access-sq9kq") pod "95181b1e-95b2-4358-bfb7-7257e9baa6d9" (UID: "95181b1e-95b2-4358-bfb7-7257e9baa6d9"). InnerVolumeSpecName "kube-api-access-sq9kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:11:46.437185 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.437144    2555 generic.go:356] "Generic (PLEG): container finished" podID="95181b1e-95b2-4358-bfb7-7257e9baa6d9" containerID="f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2" exitCode=0
Jul 29 20:11:46.437185 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.437182    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"
Jul 29 20:11:46.437436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.437229    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" event={"ID":"95181b1e-95b2-4358-bfb7-7257e9baa6d9","Type":"ContainerDied","Data":"f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2"}
Jul 29 20:11:46.437436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.437318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4" event={"ID":"95181b1e-95b2-4358-bfb7-7257e9baa6d9","Type":"ContainerDied","Data":"032d98a095738a403771a9db634795ce2b201dad5bb7b5466ddd30462edb6dbd"}
Jul 29 20:11:46.437436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.437334    2555 scope.go:117] "RemoveContainer" containerID="f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2"
Jul 29 20:11:46.438840 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.438813    2555 generic.go:356] "Generic (PLEG): container finished" podID="3045d486-0a19-40ed-87b4-de63ffc1e9d3" containerID="7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21" exitCode=0
Jul 29 20:11:46.439114 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.438874    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"
Jul 29 20:11:46.439114 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.438881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf" event={"ID":"3045d486-0a19-40ed-87b4-de63ffc1e9d3","Type":"ContainerDied","Data":"7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21"}
Jul 29 20:11:46.439114 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.438924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf" event={"ID":"3045d486-0a19-40ed-87b4-de63ffc1e9d3","Type":"ContainerDied","Data":"df23fe27a66eb6236e3e1d31befff158368b6ce9fdd82400640f79c670b34c62"}
Jul 29 20:11:46.447528 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.447485    2555 scope.go:117] "RemoveContainer" containerID="f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2"
Jul 29 20:11:46.447950 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:11:46.447804    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2\": container with ID starting with f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2 not found: ID does not exist" containerID="f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2"
Jul 29 20:11:46.447950 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.447838    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2"} err="failed to get container status \"f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2\": rpc error: code = NotFound desc = could not find container \"f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2\": container with ID starting with f67830714ebee99b56736b118863fb86eb281df8826cd46c8bd09a90cee135a2 not found: ID does not exist"
Jul 29 20:11:46.447950 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.447862    2555 scope.go:117] "RemoveContainer" containerID="7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21"
Jul 29 20:11:46.456349 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.456332    2555 scope.go:117] "RemoveContainer" containerID="7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21"
Jul 29 20:11:46.456653 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:11:46.456597    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21\": container with ID starting with 7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21 not found: ID does not exist" containerID="7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21"
Jul 29 20:11:46.456653 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.456624    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21"} err="failed to get container status \"7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21\": rpc error: code = NotFound desc = could not find container \"7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21\": container with ID starting with 7fbf3cb04cd09361c85860cf813fd909124ecd9362a4e71ea2ce2011d99a7c21 not found: ID does not exist"
Jul 29 20:11:46.463162 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.463141    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"]
Jul 29 20:11:46.468091 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.468067    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78b35e-86d48c559d-w5cc4"]
Jul 29 20:11:46.472750 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.472730    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95181b1e-95b2-4358-bfb7-7257e9baa6d9-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:11:46.472825 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.472752    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxh9r\" (UniqueName: \"kubernetes.io/projected/3045d486-0a19-40ed-87b4-de63ffc1e9d3-kube-api-access-qxh9r\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:11:46.472825 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.472763    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq9kq\" (UniqueName: \"kubernetes.io/projected/95181b1e-95b2-4358-bfb7-7257e9baa6d9-kube-api-access-sq9kq\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:11:46.478156 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.478132    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"]
Jul 29 20:11:46.480445 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.480425    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-78b35e-5d84899d48-lf7jf"]
Jul 29 20:11:46.658878 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.658844    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3045d486-0a19-40ed-87b4-de63ffc1e9d3" path="/var/lib/kubelet/pods/3045d486-0a19-40ed-87b4-de63ffc1e9d3/volumes"
Jul 29 20:11:46.659238 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:11:46.659224    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95181b1e-95b2-4358-bfb7-7257e9baa6d9" path="/var/lib/kubelet/pods/95181b1e-95b2-4358-bfb7-7257e9baa6d9/volumes"
Jul 29 20:12:08.762817 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:08.762775    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"]
Jul 29 20:12:08.763303 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:08.762985    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" podUID="06613625-c97c-4440-a3ff-c268dd3a3ba1" containerName="maas-api" containerID="cri-o://7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783" gracePeriod=30
Jul 29 20:12:08.830210 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:08.830173    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"]
Jul 29 20:12:08.830510 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:08.830456    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg" podUID="31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" containerName="payload-processing" containerID="cri-o://508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58" gracePeriod=30
Jul 29 20:12:09.105572 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.105548    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:12:09.128054 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.128025    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:12:09.282247 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282213    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls\") pod \"06613625-c97c-4440-a3ff-c268dd3a3ba1\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") "
Jul 29 20:12:09.282436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282301    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06613625-c97c-4440-a3ff-c268dd3a3ba1-openshift-service-ca\") pod \"06613625-c97c-4440-a3ff-c268dd3a3ba1\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") "
Jul 29 20:12:09.282436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282364    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-plugins-config-volume\") pod \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") "
Jul 29 20:12:09.282436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282423    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qmwr\" (UniqueName: \"kubernetes.io/projected/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-kube-api-access-5qmwr\") pod \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\" (UID: \"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3\") "
Jul 29 20:12:09.282629 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282455    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzfv8\" (UniqueName: \"kubernetes.io/projected/06613625-c97c-4440-a3ff-c268dd3a3ba1-kube-api-access-wzfv8\") pod \"06613625-c97c-4440-a3ff-c268dd3a3ba1\" (UID: \"06613625-c97c-4440-a3ff-c268dd3a3ba1\") "
Jul 29 20:12:09.282745 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282713    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06613625-c97c-4440-a3ff-c268dd3a3ba1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06613625-c97c-4440-a3ff-c268dd3a3ba1" (UID: "06613625-c97c-4440-a3ff-c268dd3a3ba1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:09.282798 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.282729    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" (UID: "31d46a95-2dd6-4ff3-a475-1fbf07e98bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:12:09.284485 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.284463    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06613625-c97c-4440-a3ff-c268dd3a3ba1" (UID: "06613625-c97c-4440-a3ff-c268dd3a3ba1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:12:09.284620 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.284603    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-kube-api-access-5qmwr" (OuterVolumeSpecName: "kube-api-access-5qmwr") pod "31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" (UID: "31d46a95-2dd6-4ff3-a475-1fbf07e98bd3"). InnerVolumeSpecName "kube-api-access-5qmwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:09.284667 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.284602    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06613625-c97c-4440-a3ff-c268dd3a3ba1-kube-api-access-wzfv8" (OuterVolumeSpecName: "kube-api-access-wzfv8") pod "06613625-c97c-4440-a3ff-c268dd3a3ba1" (UID: "06613625-c97c-4440-a3ff-c268dd3a3ba1"). InnerVolumeSpecName "kube-api-access-wzfv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:12:09.383937 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.383860    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06613625-c97c-4440-a3ff-c268dd3a3ba1-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:12:09.383937 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.383889    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:12:09.383937 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.383899    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmwr\" (UniqueName: \"kubernetes.io/projected/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3-kube-api-access-5qmwr\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:12:09.383937 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.383907    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfv8\" (UniqueName: \"kubernetes.io/projected/06613625-c97c-4440-a3ff-c268dd3a3ba1-kube-api-access-wzfv8\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:12:09.383937 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.383918    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06613625-c97c-4440-a3ff-c268dd3a3ba1-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:12:09.554366 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.554334    2555 generic.go:356] "Generic (PLEG): container finished" podID="31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" containerID="508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58" exitCode=0
Jul 29 20:12:09.554557 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.554379    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"
Jul 29 20:12:09.554557 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.554426    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg" event={"ID":"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3","Type":"ContainerDied","Data":"508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58"}
Jul 29 20:12:09.554557 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.554489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg" event={"ID":"31d46a95-2dd6-4ff3-a475-1fbf07e98bd3","Type":"ContainerDied","Data":"e2d2b64c96fd87729e8d5af08dfb9f8be97695536e33982c9cb55bcf0fa61f26"}
Jul 29 20:12:09.554557 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.554519    2555 scope.go:117] "RemoveContainer" containerID="508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58"
Jul 29 20:12:09.555945 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.555913    2555 generic.go:356] "Generic (PLEG): container finished" podID="06613625-c97c-4440-a3ff-c268dd3a3ba1" containerID="7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783" exitCode=0
Jul 29 20:12:09.556058 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.555956    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"
Jul 29 20:12:09.556058 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.555960    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" event={"ID":"06613625-c97c-4440-a3ff-c268dd3a3ba1","Type":"ContainerDied","Data":"7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783"}
Jul 29 20:12:09.556058 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.556017    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v" event={"ID":"06613625-c97c-4440-a3ff-c268dd3a3ba1","Type":"ContainerDied","Data":"2604df97f803ab36ff749e5392a0ceaa69228d1276372dd59674b071666b8c20"}
Jul 29 20:12:09.564188 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.564171    2555 scope.go:117] "RemoveContainer" containerID="508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58"
Jul 29 20:12:09.564472 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:12:09.564449    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58\": container with ID starting with 508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58 not found: ID does not exist" containerID="508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58"
Jul 29 20:12:09.564521 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.564486    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58"} err="failed to get container status \"508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58\": rpc error: code = NotFound desc = could not find container \"508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58\": container with ID starting with 508e2a71f97cef14b23897d5b528951245c1800f569fe5c64f4b43cc4b1a6a58 not found: ID does not exist"
Jul 29 20:12:09.564521 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.564506    2555 scope.go:117] "RemoveContainer" containerID="7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783"
Jul 29 20:12:09.572440 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.572419    2555 scope.go:117] "RemoveContainer" containerID="7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783"
Jul 29 20:12:09.572742 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:12:09.572722    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783\": container with ID starting with 7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783 not found: ID does not exist" containerID="7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783"
Jul 29 20:12:09.572810 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.572749    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783"} err="failed to get container status \"7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783\": rpc error: code = NotFound desc = could not find container \"7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783\": container with ID starting with 7f585f9091c75677ae76428287c4b4d55fd2397a186f428c42191d90abc28783 not found: ID does not exist"
Jul 29 20:12:09.575983 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.575961    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"]
Jul 29 20:12:09.580311 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.580291    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d3ceb2-8665ccb9bd-rbxtg"]
Jul 29 20:12:09.590705 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.590679    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"]
Jul 29 20:12:09.593019 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:09.592999    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3ceb2-786bff758b-tt28v"]
Jul 29 20:12:10.658661 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:10.658621    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06613625-c97c-4440-a3ff-c268dd3a3ba1" path="/var/lib/kubelet/pods/06613625-c97c-4440-a3ff-c268dd3a3ba1/volumes"
Jul 29 20:12:10.659177 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:10.659152    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" path="/var/lib/kubelet/pods/31d46a95-2dd6-4ff3-a475-1fbf07e98bd3/volumes"
Jul 29 20:12:19.871913 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.871864    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"]
Jul 29 20:12:19.872424 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872408    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" containerName="payload-processing"
Jul 29 20:12:19.872470 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872427    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" containerName="payload-processing"
Jul 29 20:12:19.872470 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872460    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06613625-c97c-4440-a3ff-c268dd3a3ba1" containerName="maas-api"
Jul 29 20:12:19.872470 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872469    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="06613625-c97c-4440-a3ff-c268dd3a3ba1" containerName="maas-api"
Jul 29 20:12:19.872602 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872490    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95181b1e-95b2-4358-bfb7-7257e9baa6d9" containerName="maas-api"
Jul 29 20:12:19.872602 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872499    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="95181b1e-95b2-4358-bfb7-7257e9baa6d9" containerName="maas-api"
Jul 29 20:12:19.872602 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872509    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3045d486-0a19-40ed-87b4-de63ffc1e9d3" containerName="payload-processing"
Jul 29 20:12:19.872602 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872517    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3045d486-0a19-40ed-87b4-de63ffc1e9d3" containerName="payload-processing"
Jul 29 20:12:19.872760 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872606    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3045d486-0a19-40ed-87b4-de63ffc1e9d3" containerName="payload-processing"
Jul 29 20:12:19.872760 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872623    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="95181b1e-95b2-4358-bfb7-7257e9baa6d9" containerName="maas-api"
Jul 29 20:12:19.872760 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872635    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="06613625-c97c-4440-a3ff-c268dd3a3ba1" containerName="maas-api"
Jul 29 20:12:19.872760 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.872645    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d46a95-2dd6-4ff3-a475-1fbf07e98bd3" containerName="payload-processing"
Jul 29 20:12:19.877772 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.877747    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:19.879847 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.879815    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7d4a80-serving-cert\""
Jul 29 20:12:19.879990 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.879822    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wzkl2\""
Jul 29 20:12:19.887348 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.886008    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"]
Jul 29 20:12:19.975053 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.975012    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vsw\" (UniqueName: \"kubernetes.io/projected/8013b929-27b9-4574-a68b-2d2c4f59a10e-kube-api-access-94vsw\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:19.975225 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.975059    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:19.975225 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:19.975188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8013b929-27b9-4574-a68b-2d2c4f59a10e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.075803 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.075771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8013b929-27b9-4574-a68b-2d2c4f59a10e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.075803 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.075813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vsw\" (UniqueName: \"kubernetes.io/projected/8013b929-27b9-4574-a68b-2d2c4f59a10e-kube-api-access-94vsw\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.076140 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.075852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.076140 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:12:20.075988    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-serving-cert: secret "maas-api-e2e-shared-a-7d4a80-serving-cert" not found
Jul 29 20:12:20.076140 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:12:20.076060    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls podName:8013b929-27b9-4574-a68b-2d2c4f59a10e nodeName:}" failed. No retries permitted until 2026-07-29 20:12:20.576038897 +0000 UTC m=+2546.534533769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls") pod "maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" (UID: "8013b929-27b9-4574-a68b-2d2c4f59a10e") : secret "maas-api-e2e-shared-a-7d4a80-serving-cert" not found
Jul 29 20:12:20.076457 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.076440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8013b929-27b9-4574-a68b-2d2c4f59a10e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.088341 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.088309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vsw\" (UniqueName: \"kubernetes.io/projected/8013b929-27b9-4574-a68b-2d2c4f59a10e-kube-api-access-94vsw\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.580442 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.580403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.583287 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.583256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls\") pod \"maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.632638 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.632603    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"]
Jul 29 20:12:20.636067 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.636041    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.637835 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.637814    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-7d4a80\""
Jul 29 20:12:20.637972 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.637885    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-7d4a80-dockercfg-9gcsw\""
Jul 29 20:12:20.647747 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.647716    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"]
Jul 29 20:12:20.680902 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.680866    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshwz\" (UniqueName: \"kubernetes.io/projected/83a059d6-879c-411e-a317-63d7185089a9-kube-api-access-qshwz\") pod \"payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.681067 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.680966    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a059d6-879c-411e-a317-63d7185089a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.689935 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.689903    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"]
Jul 29 20:12:20.693385 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.693361    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.705017 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.704992    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"]
Jul 29 20:12:20.782471 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.782436    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xxw\" (UniqueName: \"kubernetes.io/projected/5c49992e-de73-4a27-9bac-59f679ac5b9c-kube-api-access-26xxw\") pod \"payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.782682 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.782483    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a059d6-879c-411e-a317-63d7185089a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.782682 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.782508    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshwz\" (UniqueName: \"kubernetes.io/projected/83a059d6-879c-411e-a317-63d7185089a9-kube-api-access-qshwz\") pod \"payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.782682 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.782530    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c49992e-de73-4a27-9bac-59f679ac5b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.783288 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.783264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a059d6-879c-411e-a317-63d7185089a9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.790896 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.790869    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:20.791207 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.791187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshwz\" (UniqueName: \"kubernetes.io/projected/83a059d6-879c-411e-a317-63d7185089a9-kube-api-access-qshwz\") pod \"payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:20.883120 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.883080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c49992e-de73-4a27-9bac-59f679ac5b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.885536 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.883218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xxw\" (UniqueName: \"kubernetes.io/projected/5c49992e-de73-4a27-9bac-59f679ac5b9c-kube-api-access-26xxw\") pod \"payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.885536 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.884362    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c49992e-de73-4a27-9bac-59f679ac5b9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.892998 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.892970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xxw\" (UniqueName: \"kubernetes.io/projected/5c49992e-de73-4a27-9bac-59f679ac5b9c-kube-api-access-26xxw\") pod \"payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:20.931890 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.931864    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"]
Jul 29 20:12:20.934261 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:12:20.934231    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8013b929_27b9_4574_a68b_2d2c4f59a10e.slice/crio-96ba2a88d5251ef1e8d9d32ff1a9a78495e58ed18ded84a7d5b50dc44ecda8d5 WatchSource:0}: Error finding container 96ba2a88d5251ef1e8d9d32ff1a9a78495e58ed18ded84a7d5b50dc44ecda8d5: Status 404 returned error can't find the container with id 96ba2a88d5251ef1e8d9d32ff1a9a78495e58ed18ded84a7d5b50dc44ecda8d5
Jul 29 20:12:20.955738 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:20.955712    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:21.005304 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.005267    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:21.109876 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.109800    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"]
Jul 29 20:12:21.111969 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:12:21.111936    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a059d6_879c_411e_a317_63d7185089a9.slice/crio-39aa858494730a1d4a039992cac3b0a45e17b0b7449ab9fca62596393c8e66fa WatchSource:0}: Error finding container 39aa858494730a1d4a039992cac3b0a45e17b0b7449ab9fca62596393c8e66fa: Status 404 returned error can't find the container with id 39aa858494730a1d4a039992cac3b0a45e17b0b7449ab9fca62596393c8e66fa
Jul 29 20:12:21.163628 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.163546    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"]
Jul 29 20:12:21.167704 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:12:21.167655    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c49992e_de73_4a27_9bac_59f679ac5b9c.slice/crio-42d100b7999c992375b8b462ffd3617e9be76069ab37fcda0b550ee11274d149 WatchSource:0}: Error finding container 42d100b7999c992375b8b462ffd3617e9be76069ab37fcda0b550ee11274d149: Status 404 returned error can't find the container with id 42d100b7999c992375b8b462ffd3617e9be76069ab37fcda0b550ee11274d149
Jul 29 20:12:21.614724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.614638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77" event={"ID":"83a059d6-879c-411e-a317-63d7185089a9","Type":"ContainerStarted","Data":"e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24"}
Jul 29 20:12:21.614724 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.614698    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77" event={"ID":"83a059d6-879c-411e-a317-63d7185089a9","Type":"ContainerStarted","Data":"39aa858494730a1d4a039992cac3b0a45e17b0b7449ab9fca62596393c8e66fa"}
Jul 29 20:12:21.614961 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.614835    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:21.616282 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.616258    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz" event={"ID":"5c49992e-de73-4a27-9bac-59f679ac5b9c","Type":"ContainerStarted","Data":"e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8"}
Jul 29 20:12:21.616421 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.616297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz" event={"ID":"5c49992e-de73-4a27-9bac-59f679ac5b9c","Type":"ContainerStarted","Data":"42d100b7999c992375b8b462ffd3617e9be76069ab37fcda0b550ee11274d149"}
Jul 29 20:12:21.616421 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.616324    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:21.617904 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.617875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" event={"ID":"8013b929-27b9-4574-a68b-2d2c4f59a10e","Type":"ContainerStarted","Data":"fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed"}
Jul 29 20:12:21.618037 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.617920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" event={"ID":"8013b929-27b9-4574-a68b-2d2c4f59a10e","Type":"ContainerStarted","Data":"96ba2a88d5251ef1e8d9d32ff1a9a78495e58ed18ded84a7d5b50dc44ecda8d5"}
Jul 29 20:12:21.618037 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.618009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:21.630018 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.629977    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77" podStartSLOduration=1.6299638509999999 podStartE2EDuration="1.629963851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:12:21.629303836 +0000 UTC m=+2547.587798753" watchObservedRunningTime="2026-07-29 20:12:21.629963851 +0000 UTC m=+2547.588458745"
Jul 29 20:12:21.645425 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.645367    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" podStartSLOduration=2.2279231250000002 podStartE2EDuration="2.645349779s" totalImagesPullingTime="417.426654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:12:21.643996898 +0000 UTC m=+2547.602491812" watchObservedRunningTime="2026-07-29 20:12:21.645349779 +0000 UTC m=+2547.603844675"
Jul 29 20:12:21.659797 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:21.659753    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz" podStartSLOduration=1.659738204 podStartE2EDuration="1.659738204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:12:21.658941444 +0000 UTC m=+2547.617436339" watchObservedRunningTime="2026-07-29 20:12:21.659738204 +0000 UTC m=+2547.618233135"
Jul 29 20:12:27.629108 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:27.629077    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:12:32.625389 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:32.625355    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:12:32.625873 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:32.625461    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:12:48.021498 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.021463    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"]
Jul 29 20:12:48.025568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.025545    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.027657 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.027632    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1a1eaa-serving-cert\""
Jul 29 20:12:48.037691 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.036779    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"]
Jul 29 20:12:48.132710 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.132675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d182f2-dbfa-4186-9377-69ba6d0c27aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.132710 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.132718    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsc8w\" (UniqueName: \"kubernetes.io/projected/38d182f2-dbfa-4186-9377-69ba6d0c27aa-kube-api-access-nsc8w\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.132929 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.132785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d182f2-dbfa-4186-9377-69ba6d0c27aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.233816 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.233781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d182f2-dbfa-4186-9377-69ba6d0c27aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.234022 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.233828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsc8w\" (UniqueName: \"kubernetes.io/projected/38d182f2-dbfa-4186-9377-69ba6d0c27aa-kube-api-access-nsc8w\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.234022 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.233877    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d182f2-dbfa-4186-9377-69ba6d0c27aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.234502 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.234482    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d182f2-dbfa-4186-9377-69ba6d0c27aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.236700 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.236674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d182f2-dbfa-4186-9377-69ba6d0c27aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.245639 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.245606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsc8w\" (UniqueName: \"kubernetes.io/projected/38d182f2-dbfa-4186-9377-69ba6d0c27aa-kube-api-access-nsc8w\") pod \"maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.340728 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.340629    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:48.484837 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.484809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"]
Jul 29 20:12:48.487233 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:12:48.487202    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d182f2_dbfa_4186_9377_69ba6d0c27aa.slice/crio-b77105d1e6d6b55c247bacd6e94d1f540ab3631351a4051b637719ecda868cef WatchSource:0}: Error finding container b77105d1e6d6b55c247bacd6e94d1f540ab3631351a4051b637719ecda868cef: Status 404 returned error can't find the container with id b77105d1e6d6b55c247bacd6e94d1f540ab3631351a4051b637719ecda868cef
Jul 29 20:12:48.755467 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.755427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5" event={"ID":"38d182f2-dbfa-4186-9377-69ba6d0c27aa","Type":"ContainerStarted","Data":"b77105d1e6d6b55c247bacd6e94d1f540ab3631351a4051b637719ecda868cef"}
Jul 29 20:12:48.818833 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.818796    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"]
Jul 29 20:12:48.824273 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.824247    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:48.826336 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.826313    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-1a1eaa\""
Jul 29 20:12:48.826449 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.826319    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-1a1eaa-dockercfg-6hnn2\""
Jul 29 20:12:48.830229 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.830206    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"]
Jul 29 20:12:48.940120 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.940082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b25r\" (UniqueName: \"kubernetes.io/projected/53336460-7a6b-4737-84f6-06248ed5eb4c-kube-api-access-8b25r\") pod \"payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:48.940315 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:48.940246    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53336460-7a6b-4737-84f6-06248ed5eb4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.041383 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.041337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b25r\" (UniqueName: \"kubernetes.io/projected/53336460-7a6b-4737-84f6-06248ed5eb4c-kube-api-access-8b25r\") pod \"payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.041863 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.041499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53336460-7a6b-4737-84f6-06248ed5eb4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.042284 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.042261    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53336460-7a6b-4737-84f6-06248ed5eb4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.049488 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.049458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b25r\" (UniqueName: \"kubernetes.io/projected/53336460-7a6b-4737-84f6-06248ed5eb4c-kube-api-access-8b25r\") pod \"payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.140458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.140413    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.284951 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.284308    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"]
Jul 29 20:12:49.287237 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:12:49.287207    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53336460_7a6b_4737_84f6_06248ed5eb4c.slice/crio-84196bbfe2dc85f8c7d90905d3ab3c671329c1033ac0afe0f3e7e030128c08fa WatchSource:0}: Error finding container 84196bbfe2dc85f8c7d90905d3ab3c671329c1033ac0afe0f3e7e030128c08fa: Status 404 returned error can't find the container with id 84196bbfe2dc85f8c7d90905d3ab3c671329c1033ac0afe0f3e7e030128c08fa
Jul 29 20:12:49.762324 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.762285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67" event={"ID":"53336460-7a6b-4737-84f6-06248ed5eb4c","Type":"ContainerStarted","Data":"2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a"}
Jul 29 20:12:49.762529 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.762353    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67" event={"ID":"53336460-7a6b-4737-84f6-06248ed5eb4c","Type":"ContainerStarted","Data":"84196bbfe2dc85f8c7d90905d3ab3c671329c1033ac0afe0f3e7e030128c08fa"}
Jul 29 20:12:49.762529 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.762387    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:12:49.764047 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.764024    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5" event={"ID":"38d182f2-dbfa-4186-9377-69ba6d0c27aa","Type":"ContainerStarted","Data":"903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f"}
Jul 29 20:12:49.764160 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.764075    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:12:49.780971 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.780921    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67" podStartSLOduration=1.7809053860000001 podStartE2EDuration="1.780905386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:12:49.779960464 +0000 UTC m=+2575.738455374" watchObservedRunningTime="2026-07-29 20:12:49.780905386 +0000 UTC m=+2575.739400280"
Jul 29 20:12:49.803693 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:49.803644    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5" podStartSLOduration=1.334609676 podStartE2EDuration="1.803627494s" totalImagesPullingTime="469.017818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:12:49.803067601 +0000 UTC m=+2575.761562518" watchObservedRunningTime="2026-07-29 20:12:49.803627494 +0000 UTC m=+2575.762122388"
Jul 29 20:12:55.774563 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:12:55.774531    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:13:00.771335 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:00.771303    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:13:17.869534 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.869442    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"]
Jul 29 20:13:17.872986 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.872964    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:17.875741 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.875708    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7d4a80\"/\"default-dockercfg-hgzvx\""
Jul 29 20:13:17.875863 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.875745    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7d4a80\"/\"openshift-service-ca.crt\""
Jul 29 20:13:17.875863 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.875781    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7d4a80\"/\"kube-root-ca.crt\""
Jul 29 20:13:17.875863 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.875792    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7d4a80\"/\"auth-test-model-7d4a80-kserve-self-signed-certs\""
Jul 29 20:13:17.883220 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.882624    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"]
Jul 29 20:13:17.913940 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.913911    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-model-cache\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:17.914087 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.913994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-dshm\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:17.914144 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.914097    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-home\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:17.914144 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.914128    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5t2p\" (UniqueName: \"kubernetes.io/projected/d655dd49-8c8d-49c7-8acd-ca59578900a0-kube-api-access-r5t2p\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:17.914236 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.914170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-tmp-dir\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:17.914236 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:17.914208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d655dd49-8c8d-49c7-8acd-ca59578900a0-tls-certs\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.015548 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.015502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-dshm\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.015758 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.015568    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-home\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.015758 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.015633    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5t2p\" (UniqueName: \"kubernetes.io/projected/d655dd49-8c8d-49c7-8acd-ca59578900a0-kube-api-access-r5t2p\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.015758 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.015667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-tmp-dir\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.015758 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.015686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d655dd49-8c8d-49c7-8acd-ca59578900a0-tls-certs\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.015758 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.015740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-model-cache\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.016063 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.016041    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-home\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.016122 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.016070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-model-cache\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.016122 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.016106    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-tmp-dir\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.018062 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.018031    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-dshm\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.018248 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.018232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d655dd49-8c8d-49c7-8acd-ca59578900a0-tls-certs\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.024516 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.024492    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5t2p\" (UniqueName: \"kubernetes.io/projected/d655dd49-8c8d-49c7-8acd-ca59578900a0-kube-api-access-r5t2p\") pod \"auth-test-model-7d4a80-kserve-5665d4f495-2n58z\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") " pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.186096 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.186071    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:18.327787 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.327761    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"]
Jul 29 20:13:18.330425 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:13:18.330393    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd655dd49_8c8d_49c7_8acd_ca59578900a0.slice/crio-f7641f846a3165e11d59c98e7211e8bb1b6eb16d77bb2297f2707bf9bb7d39b6 WatchSource:0}: Error finding container f7641f846a3165e11d59c98e7211e8bb1b6eb16d77bb2297f2707bf9bb7d39b6: Status 404 returned error can't find the container with id f7641f846a3165e11d59c98e7211e8bb1b6eb16d77bb2297f2707bf9bb7d39b6
Jul 29 20:13:18.912537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.912501    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z" event={"ID":"d655dd49-8c8d-49c7-8acd-ca59578900a0","Type":"ContainerStarted","Data":"1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f"}
Jul 29 20:13:18.912537 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.912543    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z" event={"ID":"d655dd49-8c8d-49c7-8acd-ca59578900a0","Type":"ContainerStarted","Data":"f7641f846a3165e11d59c98e7211e8bb1b6eb16d77bb2297f2707bf9bb7d39b6"}
Jul 29 20:13:18.932992 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:18.932946    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z" podStartSLOduration=1.932932463 podStartE2EDuration="1.932932463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:13:18.931712989 +0000 UTC m=+2604.890207895" watchObservedRunningTime="2026-07-29 20:13:18.932932463 +0000 UTC m=+2604.891427356"
Jul 29 20:13:28.186797 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:28.186764    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:28.187301 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:28.186811    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:28.199236 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:28.199205    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:13:28.975178 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:13:28.975151    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:14:54.718508 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:14:54.718483    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:14:54.722375 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:14:54.722354    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:18:21.171425 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:21.171392    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"]
Jul 29 20:18:25.980401 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:25.980366    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"]
Jul 29 20:18:25.984108 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:25.984088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:25.986126 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:25.986108    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7d4a80\"/\"sub-test-model-7d4a80-kserve-self-signed-certs\""
Jul 29 20:18:25.995106 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:25.995081    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"]
Jul 29 20:18:26.022170 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.022135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-model-cache\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.022310 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.022174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-home\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.022310 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.022204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-dshm\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.022310 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.022291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tmp-dir\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.022456 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.022355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtvtb\" (UniqueName: \"kubernetes.io/projected/e3dd3f96-d888-46ad-acd1-7d4446c1640d-kube-api-access-wtvtb\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.022456 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.022394    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tls-certs\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123404 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-model-cache\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123404 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123408    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-home\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123697 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123431    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-dshm\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123697 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123479    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tmp-dir\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123697 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123533    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvtb\" (UniqueName: \"kubernetes.io/projected/e3dd3f96-d888-46ad-acd1-7d4446c1640d-kube-api-access-wtvtb\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123697 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123564    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tls-certs\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123925 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-model-cache\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.123925 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123914    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-home\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.124039 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.123956    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tmp-dir\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.126063 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.126041    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-dshm\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.126339 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.126322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tls-certs\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.131763 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.131740    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvtb\" (UniqueName: \"kubernetes.io/projected/e3dd3f96-d888-46ad-acd1-7d4446c1640d-kube-api-access-wtvtb\") pod \"sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") " pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.294327 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.294237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:26.433359 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:26.433335    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"]
Jul 29 20:18:26.435633 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:18:26.435602    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3dd3f96_d888_46ad_acd1_7d4446c1640d.slice/crio-9cf48705d76a032e7fe1a75902fbd8922f2d78af9217fd14a239bdff6d7c9b11 WatchSource:0}: Error finding container 9cf48705d76a032e7fe1a75902fbd8922f2d78af9217fd14a239bdff6d7c9b11: Status 404 returned error can't find the container with id 9cf48705d76a032e7fe1a75902fbd8922f2d78af9217fd14a239bdff6d7c9b11
Jul 29 20:18:27.429082 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:27.429042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w" event={"ID":"e3dd3f96-d888-46ad-acd1-7d4446c1640d","Type":"ContainerStarted","Data":"01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282"}
Jul 29 20:18:27.429547 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:27.429091    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w" event={"ID":"e3dd3f96-d888-46ad-acd1-7d4446c1640d","Type":"ContainerStarted","Data":"9cf48705d76a032e7fe1a75902fbd8922f2d78af9217fd14a239bdff6d7c9b11"}
Jul 29 20:18:27.445588 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:27.445535    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w" podStartSLOduration=2.445520605 podStartE2EDuration="2.445520605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:18:27.444841769 +0000 UTC m=+2913.403336660" watchObservedRunningTime="2026-07-29 20:18:27.445520605 +0000 UTC m=+2913.404015505"
Jul 29 20:18:36.191096 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.191040    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z" podUID="d655dd49-8c8d-49c7-8acd-ca59578900a0" containerName="main" containerID="cri-o://1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f" gracePeriod=45
Jul 29 20:18:36.294703 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.294666    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:36.294887 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.294729    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:36.307649 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.307618    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:36.455564 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.455502    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:18:36.479482 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.479454    2555 generic.go:356] "Generic (PLEG): container finished" podID="d655dd49-8c8d-49c7-8acd-ca59578900a0" containerID="1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f" exitCode=0
Jul 29 20:18:36.479665 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.479498    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"
Jul 29 20:18:36.479665 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.479546    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z" event={"ID":"d655dd49-8c8d-49c7-8acd-ca59578900a0","Type":"ContainerDied","Data":"1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f"}
Jul 29 20:18:36.479665 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.479623    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z" event={"ID":"d655dd49-8c8d-49c7-8acd-ca59578900a0","Type":"ContainerDied","Data":"f7641f846a3165e11d59c98e7211e8bb1b6eb16d77bb2297f2707bf9bb7d39b6"}
Jul 29 20:18:36.479665 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.479655    2555 scope.go:117] "RemoveContainer" containerID="1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f"
Jul 29 20:18:36.488345 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.488316    2555 scope.go:117] "RemoveContainer" containerID="1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f"
Jul 29 20:18:36.488624 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:18:36.488605    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f\": container with ID starting with 1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f not found: ID does not exist" containerID="1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f"
Jul 29 20:18:36.488690 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.488632    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f"} err="failed to get container status \"1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f\": rpc error: code = NotFound desc = could not find container \"1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f\": container with ID starting with 1a7a24ec4aa4dd166d0342bf2c853f2730011d7c1f74d4d8d5151d30ef81a43f not found: ID does not exist"
Jul 29 20:18:36.497009 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.496984    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:18:36.521228 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521207    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-home\") pod \"d655dd49-8c8d-49c7-8acd-ca59578900a0\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") "
Jul 29 20:18:36.521362 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521261    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5t2p\" (UniqueName: \"kubernetes.io/projected/d655dd49-8c8d-49c7-8acd-ca59578900a0-kube-api-access-r5t2p\") pod \"d655dd49-8c8d-49c7-8acd-ca59578900a0\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") "
Jul 29 20:18:36.521362 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521280    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-model-cache\") pod \"d655dd49-8c8d-49c7-8acd-ca59578900a0\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") "
Jul 29 20:18:36.521362 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521319    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-tmp-dir\") pod \"d655dd49-8c8d-49c7-8acd-ca59578900a0\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") "
Jul 29 20:18:36.521539 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521386    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-dshm\") pod \"d655dd49-8c8d-49c7-8acd-ca59578900a0\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") "
Jul 29 20:18:36.521539 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521426    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d655dd49-8c8d-49c7-8acd-ca59578900a0-tls-certs\") pod \"d655dd49-8c8d-49c7-8acd-ca59578900a0\" (UID: \"d655dd49-8c8d-49c7-8acd-ca59578900a0\") "
Jul 29 20:18:36.521539 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521484    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-home" (OuterVolumeSpecName: "home") pod "d655dd49-8c8d-49c7-8acd-ca59578900a0" (UID: "d655dd49-8c8d-49c7-8acd-ca59578900a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:18:36.521706 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521629    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d655dd49-8c8d-49c7-8acd-ca59578900a0" (UID: "d655dd49-8c8d-49c7-8acd-ca59578900a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:18:36.521706 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521671    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d655dd49-8c8d-49c7-8acd-ca59578900a0" (UID: "d655dd49-8c8d-49c7-8acd-ca59578900a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:18:36.521790 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521771    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:18:36.521844 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521792    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:18:36.521844 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.521805    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:18:36.523753 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.523731    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d655dd49-8c8d-49c7-8acd-ca59578900a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d655dd49-8c8d-49c7-8acd-ca59578900a0" (UID: "d655dd49-8c8d-49c7-8acd-ca59578900a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:18:36.523975 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.523950    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d655dd49-8c8d-49c7-8acd-ca59578900a0-kube-api-access-r5t2p" (OuterVolumeSpecName: "kube-api-access-r5t2p") pod "d655dd49-8c8d-49c7-8acd-ca59578900a0" (UID: "d655dd49-8c8d-49c7-8acd-ca59578900a0"). InnerVolumeSpecName "kube-api-access-r5t2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:18:36.523975 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.523953    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-dshm" (OuterVolumeSpecName: "dshm") pod "d655dd49-8c8d-49c7-8acd-ca59578900a0" (UID: "d655dd49-8c8d-49c7-8acd-ca59578900a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:18:36.622847 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.622815    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d655dd49-8c8d-49c7-8acd-ca59578900a0-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:18:36.622847 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.622842    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d655dd49-8c8d-49c7-8acd-ca59578900a0-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:18:36.622847 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.622854    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5t2p\" (UniqueName: \"kubernetes.io/projected/d655dd49-8c8d-49c7-8acd-ca59578900a0-kube-api-access-r5t2p\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:18:36.796561 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.796463    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"]
Jul 29 20:18:36.799597 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:36.799553    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/auth-test-model-7d4a80-kserve-5665d4f495-2n58z"]
Jul 29 20:18:38.659193 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:18:38.659160    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d655dd49-8c8d-49c7-8acd-ca59578900a0" path="/var/lib/kubelet/pods/d655dd49-8c8d-49c7-8acd-ca59578900a0/volumes"
Jul 29 20:19:45.286695 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:45.286659    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"]
Jul 29 20:19:54.750908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:54.750871    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:19:54.755906 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:54.755880    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:19:59.083858 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.083823    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"]
Jul 29 20:19:59.084455 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.084435    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d655dd49-8c8d-49c7-8acd-ca59578900a0" containerName="main"
Jul 29 20:19:59.084500 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.084461    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d655dd49-8c8d-49c7-8acd-ca59578900a0" containerName="main"
Jul 29 20:19:59.084643 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.084627    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d655dd49-8c8d-49c7-8acd-ca59578900a0" containerName="main"
Jul 29 20:19:59.088415 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.088397    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.090701 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.090675    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7d4a80\"/\"rate-test-model-7d4a80-kserve-self-signed-certs\""
Jul 29 20:19:59.100608 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.100562    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"]
Jul 29 20:19:59.202386 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.202346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tls-certs\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.202541 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.202460    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tmp-dir\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.202605 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.202541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-model-cache\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.202669 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.202652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-dshm\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.202762 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.202744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sftz\" (UniqueName: \"kubernetes.io/projected/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-kube-api-access-5sftz\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.202799 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.202779    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-home\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304204 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-dshm\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304369 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sftz\" (UniqueName: \"kubernetes.io/projected/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-kube-api-access-5sftz\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304369 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-home\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304454 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tls-certs\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304454 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304443    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tmp-dir\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304564 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-model-cache\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304730 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-home\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304818 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304795    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tmp-dir\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.304872 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.304833    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-model-cache\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.306612 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.306565    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-dshm\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.306892 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.306874    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tls-certs\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.312756 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.312734    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sftz\" (UniqueName: \"kubernetes.io/projected/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-kube-api-access-5sftz\") pod \"rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") " pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.399262 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.399179    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:19:59.541649 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.541617    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"]
Jul 29 20:19:59.543176 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:19:59.543146    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95f9f71_54c8_4a67_84b2_97bc913b0dcc.slice/crio-0259d0b28ba7d5a243b7465481ca834120e440ecfde47061b9ce795e7f35c800 WatchSource:0}: Error finding container 0259d0b28ba7d5a243b7465481ca834120e440ecfde47061b9ce795e7f35c800: Status 404 returned error can't find the container with id 0259d0b28ba7d5a243b7465481ca834120e440ecfde47061b9ce795e7f35c800
Jul 29 20:19:59.891836 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.891799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm" event={"ID":"d95f9f71-54c8-4a67-84b2-97bc913b0dcc","Type":"ContainerStarted","Data":"fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2"}
Jul 29 20:19:59.892024 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.891852    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm" event={"ID":"d95f9f71-54c8-4a67-84b2-97bc913b0dcc","Type":"ContainerStarted","Data":"0259d0b28ba7d5a243b7465481ca834120e440ecfde47061b9ce795e7f35c800"}
Jul 29 20:19:59.911771 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:19:59.911722    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm" podStartSLOduration=0.911705688 podStartE2EDuration="911.705688ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:19:59.910253256 +0000 UTC m=+3005.868748171" watchObservedRunningTime="2026-07-29 20:19:59.911705688 +0000 UTC m=+3005.870200581"
Jul 29 20:20:00.306356 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.306316    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w" podUID="e3dd3f96-d888-46ad-acd1-7d4446c1640d" containerName="main" containerID="cri-o://01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282" gracePeriod=45
Jul 29 20:20:00.577527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.577501    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:20:00.718402 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718371    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tls-certs\") pod \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") "
Jul 29 20:20:00.718616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718441    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tmp-dir\") pod \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") "
Jul 29 20:20:00.718616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718460    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-model-cache\") pod \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") "
Jul 29 20:20:00.718616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718511    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-dshm\") pod \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") "
Jul 29 20:20:00.718616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718546    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtvtb\" (UniqueName: \"kubernetes.io/projected/e3dd3f96-d888-46ad-acd1-7d4446c1640d-kube-api-access-wtvtb\") pod \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") "
Jul 29 20:20:00.718616 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718570    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-home\") pod \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\" (UID: \"e3dd3f96-d888-46ad-acd1-7d4446c1640d\") "
Jul 29 20:20:00.718915 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.718769    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3dd3f96-d888-46ad-acd1-7d4446c1640d" (UID: "e3dd3f96-d888-46ad-acd1-7d4446c1640d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:20:00.719034 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.719014    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:20:00.719373 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.719008    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-home" (OuterVolumeSpecName: "home") pod "e3dd3f96-d888-46ad-acd1-7d4446c1640d" (UID: "e3dd3f96-d888-46ad-acd1-7d4446c1640d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:20:00.719459 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.719057    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3dd3f96-d888-46ad-acd1-7d4446c1640d" (UID: "e3dd3f96-d888-46ad-acd1-7d4446c1640d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:20:00.720699 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.720674    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-dshm" (OuterVolumeSpecName: "dshm") pod "e3dd3f96-d888-46ad-acd1-7d4446c1640d" (UID: "e3dd3f96-d888-46ad-acd1-7d4446c1640d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:20:00.720797 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.720727    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3dd3f96-d888-46ad-acd1-7d4446c1640d" (UID: "e3dd3f96-d888-46ad-acd1-7d4446c1640d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:20:00.720843 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.720809    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3dd3f96-d888-46ad-acd1-7d4446c1640d-kube-api-access-wtvtb" (OuterVolumeSpecName: "kube-api-access-wtvtb") pod "e3dd3f96-d888-46ad-acd1-7d4446c1640d" (UID: "e3dd3f96-d888-46ad-acd1-7d4446c1640d"). InnerVolumeSpecName "kube-api-access-wtvtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:20:00.820081 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.820001    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:20:00.820081 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.820027    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:20:00.820081 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.820036    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtvtb\" (UniqueName: \"kubernetes.io/projected/e3dd3f96-d888-46ad-acd1-7d4446c1640d-kube-api-access-wtvtb\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:20:00.820081 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.820045    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3dd3f96-d888-46ad-acd1-7d4446c1640d-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:20:00.820081 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.820052    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3dd3f96-d888-46ad-acd1-7d4446c1640d-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:20:00.898749 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.898713    2555 generic.go:356] "Generic (PLEG): container finished" podID="e3dd3f96-d888-46ad-acd1-7d4446c1640d" containerID="01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282" exitCode=0
Jul 29 20:20:00.898908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.898759    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"
Jul 29 20:20:00.898908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.898800    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w" event={"ID":"e3dd3f96-d888-46ad-acd1-7d4446c1640d","Type":"ContainerDied","Data":"01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282"}
Jul 29 20:20:00.898908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.898847    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w" event={"ID":"e3dd3f96-d888-46ad-acd1-7d4446c1640d","Type":"ContainerDied","Data":"9cf48705d76a032e7fe1a75902fbd8922f2d78af9217fd14a239bdff6d7c9b11"}
Jul 29 20:20:00.898908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.898869    2555 scope.go:117] "RemoveContainer" containerID="01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282"
Jul 29 20:20:00.908602 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.908553    2555 scope.go:117] "RemoveContainer" containerID="01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282"
Jul 29 20:20:00.908853 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:20:00.908835    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282\": container with ID starting with 01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282 not found: ID does not exist" containerID="01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282"
Jul 29 20:20:00.908903 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.908864    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282"} err="failed to get container status \"01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282\": rpc error: code = NotFound desc = could not find container \"01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282\": container with ID starting with 01bb09691707b273650658784ba9aed28123522af812e7e44424fbfc69a0e282 not found: ID does not exist"
Jul 29 20:20:00.921031 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.921001    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"]
Jul 29 20:20:00.924872 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:00.924848    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/sub-test-model-7d4a80-kserve-7b99f5f6b7-w8j8w"]
Jul 29 20:20:02.659208 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:02.659177    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3dd3f96-d888-46ad-acd1-7d4446c1640d" path="/var/lib/kubelet/pods/e3dd3f96-d888-46ad-acd1-7d4446c1640d/volumes"
Jul 29 20:20:09.400289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:09.400245    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:20:09.400738 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:09.400332    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:20:09.412743 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:09.412722    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:20:09.953393 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:20:09.953349    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:21:19.004266 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:19.004231    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"]
Jul 29 20:21:34.023937 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.023873    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm" podUID="d95f9f71-54c8-4a67-84b2-97bc913b0dcc" containerName="main" containerID="cri-o://fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2" gracePeriod=45
Jul 29 20:21:34.285247 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.285173    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:21:34.315371 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315346    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tls-certs\") pod \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") "
Jul 29 20:21:34.315527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315384    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-home\") pod \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") "
Jul 29 20:21:34.315527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315421    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-dshm\") pod \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") "
Jul 29 20:21:34.315527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315469    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-model-cache\") pod \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") "
Jul 29 20:21:34.315527 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315488    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tmp-dir\") pod \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") "
Jul 29 20:21:34.315786 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315538    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sftz\" (UniqueName: \"kubernetes.io/projected/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-kube-api-access-5sftz\") pod \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\" (UID: \"d95f9f71-54c8-4a67-84b2-97bc913b0dcc\") "
Jul 29 20:21:34.315786 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315755    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d95f9f71-54c8-4a67-84b2-97bc913b0dcc" (UID: "d95f9f71-54c8-4a67-84b2-97bc913b0dcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:21:34.315901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315771    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-home" (OuterVolumeSpecName: "home") pod "d95f9f71-54c8-4a67-84b2-97bc913b0dcc" (UID: "d95f9f71-54c8-4a67-84b2-97bc913b0dcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:21:34.315901 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.315844    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d95f9f71-54c8-4a67-84b2-97bc913b0dcc" (UID: "d95f9f71-54c8-4a67-84b2-97bc913b0dcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:21:34.316336 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.316313    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:21:34.316433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.316341    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:21:34.316433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.316379    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:21:34.317689 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.317660    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d95f9f71-54c8-4a67-84b2-97bc913b0dcc" (UID: "d95f9f71-54c8-4a67-84b2-97bc913b0dcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:21:34.317785 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.317767    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-dshm" (OuterVolumeSpecName: "dshm") pod "d95f9f71-54c8-4a67-84b2-97bc913b0dcc" (UID: "d95f9f71-54c8-4a67-84b2-97bc913b0dcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:21:34.317935 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.317912    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-kube-api-access-5sftz" (OuterVolumeSpecName: "kube-api-access-5sftz") pod "d95f9f71-54c8-4a67-84b2-97bc913b0dcc" (UID: "d95f9f71-54c8-4a67-84b2-97bc913b0dcc"). InnerVolumeSpecName "kube-api-access-5sftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:21:34.354638 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.354607    2555 generic.go:356] "Generic (PLEG): container finished" podID="d95f9f71-54c8-4a67-84b2-97bc913b0dcc" containerID="fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2" exitCode=0
Jul 29 20:21:34.354638 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.354639    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"
Jul 29 20:21:34.354827 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.354692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm" event={"ID":"d95f9f71-54c8-4a67-84b2-97bc913b0dcc","Type":"ContainerDied","Data":"fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2"}
Jul 29 20:21:34.354827 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.354733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm" event={"ID":"d95f9f71-54c8-4a67-84b2-97bc913b0dcc","Type":"ContainerDied","Data":"0259d0b28ba7d5a243b7465481ca834120e440ecfde47061b9ce795e7f35c800"}
Jul 29 20:21:34.354827 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.354751    2555 scope.go:117] "RemoveContainer" containerID="fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2"
Jul 29 20:21:34.363923 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.363905    2555 scope.go:117] "RemoveContainer" containerID="fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2"
Jul 29 20:21:34.364178 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:21:34.364159    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2\": container with ID starting with fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2 not found: ID does not exist" containerID="fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2"
Jul 29 20:21:34.364223 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.364189    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2"} err="failed to get container status \"fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2\": rpc error: code = NotFound desc = could not find container \"fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2\": container with ID starting with fae0dd8953a36bdb6b4d94aea23b019598092e74eb46362455e82d484f9648c2 not found: ID does not exist"
Jul 29 20:21:34.375058 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.375030    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"]
Jul 29 20:21:34.377787 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.377766    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7d4a80/rate-test-model-7d4a80-kserve-78fd6979d7-fmbzm"]
Jul 29 20:21:34.416984 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.416956    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sftz\" (UniqueName: \"kubernetes.io/projected/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-kube-api-access-5sftz\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:21:34.416984 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.416982    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:21:34.417126 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.416995    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d95f9f71-54c8-4a67-84b2-97bc913b0dcc-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:21:34.658848 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:34.658770    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95f9f71-54c8-4a67-84b2-97bc913b0dcc" path="/var/lib/kubelet/pods/d95f9f71-54c8-4a67-84b2-97bc913b0dcc/volumes"
Jul 29 20:21:43.458066 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458025    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"]
Jul 29 20:21:43.458462 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458423    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3dd3f96-d888-46ad-acd1-7d4446c1640d" containerName="main"
Jul 29 20:21:43.458462 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458433    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3dd3f96-d888-46ad-acd1-7d4446c1640d" containerName="main"
Jul 29 20:21:43.458462 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458453    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95f9f71-54c8-4a67-84b2-97bc913b0dcc" containerName="main"
Jul 29 20:21:43.458462 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458458    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95f9f71-54c8-4a67-84b2-97bc913b0dcc" containerName="main"
Jul 29 20:21:43.458704 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458531    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95f9f71-54c8-4a67-84b2-97bc913b0dcc" containerName="main"
Jul 29 20:21:43.458704 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.458540    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3dd3f96-d888-46ad-acd1-7d4446c1640d" containerName="main"
Jul 29 20:21:43.463079 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.463060    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.465131 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.465107    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-658cad-dockercfg-njg7f\""
Jul 29 20:21:43.465244 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.465129    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-658cad\""
Jul 29 20:21:43.472285 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.472260    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"]
Jul 29 20:21:43.502235 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.502197    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfk9\" (UniqueName: \"kubernetes.io/projected/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-kube-api-access-rqfk9\") pod \"payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.502380 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.502341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.602948 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.602897    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfk9\" (UniqueName: \"kubernetes.io/projected/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-kube-api-access-rqfk9\") pod \"payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.603121 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.603011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.603684 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.603664    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.614382 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.614349    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfk9\" (UniqueName: \"kubernetes.io/projected/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-kube-api-access-rqfk9\") pod \"payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.775643 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.775528    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:43.929305 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:43.929277    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"]
Jul 29 20:21:43.931843 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:21:43.931808    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c505fe5_b9fa_4a54_9b6a_2a5cd3d9419b.slice/crio-778bdedd11b17ec76a91e1715047c1a18caaac2437baa1c9503ddc212b7aa582 WatchSource:0}: Error finding container 778bdedd11b17ec76a91e1715047c1a18caaac2437baa1c9503ddc212b7aa582: Status 404 returned error can't find the container with id 778bdedd11b17ec76a91e1715047c1a18caaac2437baa1c9503ddc212b7aa582
Jul 29 20:21:44.401588 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:44.401553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc" event={"ID":"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b","Type":"ContainerStarted","Data":"341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c"}
Jul 29 20:21:44.401774 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:44.401628    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc" event={"ID":"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b","Type":"ContainerStarted","Data":"778bdedd11b17ec76a91e1715047c1a18caaac2437baa1c9503ddc212b7aa582"}
Jul 29 20:21:44.401774 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:44.401735    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:21:44.417232 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:44.417181    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc" podStartSLOduration=1.417164857 podStartE2EDuration="1.417164857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:21:44.415768624 +0000 UTC m=+3110.374263519" watchObservedRunningTime="2026-07-29 20:21:44.417164857 +0000 UTC m=+3110.375659751"
Jul 29 20:21:55.408799 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:21:55.408703    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:22:11.783113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.783079    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"]
Jul 29 20:22:11.786555 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.786532    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:11.788695 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.788674    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-259e5b-serving-cert\""
Jul 29 20:22:11.797652 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.797630    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"]
Jul 29 20:22:11.973690 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.973655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:11.973690 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.973687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsf9\" (UniqueName: \"kubernetes.io/projected/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-kube-api-access-xfsf9\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:11.973959 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:11.973856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.075307 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.075205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.075307 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.075275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.075307 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.075298    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsf9\" (UniqueName: \"kubernetes.io/projected/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-kube-api-access-xfsf9\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.075569 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:22:12.075361    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-serving-cert: secret "maas-api-e2e-ipp-b-259e5b-serving-cert" not found
Jul 29 20:22:12.075569 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:22:12.075447    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls podName:c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493 nodeName:}" failed. No retries permitted until 2026-07-29 20:22:12.575422871 +0000 UTC m=+3138.533917756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls") pod "maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" (UID: "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493") : secret "maas-api-e2e-ipp-b-259e5b-serving-cert" not found
Jul 29 20:22:12.076108 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.076084    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.085326 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.085301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsf9\" (UniqueName: \"kubernetes.io/projected/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-kube-api-access-xfsf9\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.541632 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.541595    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"]
Jul 29 20:22:12.545835 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.545813    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.547837 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.547813    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-259e5b-dockercfg-zhp6k\""
Jul 29 20:22:12.547949 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.547822    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-259e5b\""
Jul 29 20:22:12.555534 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.555509    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"]
Jul 29 20:22:12.579677 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.579636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:12.579843 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:22:12.579784    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-serving-cert: secret "maas-api-e2e-ipp-b-259e5b-serving-cert" not found
Jul 29 20:22:12.579913 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:22:12.579854    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls podName:c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493 nodeName:}" failed. No retries permitted until 2026-07-29 20:22:13.579835157 +0000 UTC m=+3139.538330028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls") pod "maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" (UID: "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493") : secret "maas-api-e2e-ipp-b-259e5b-serving-cert" not found
Jul 29 20:22:12.595289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.595259    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"]
Jul 29 20:22:12.598652 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.598637    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.606966 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.606943    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"]
Jul 29 20:22:12.680371 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.680331    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1bc531-5c5d-43f7-a061-60ff3c59d148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.680541 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.680488    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvgv\" (UniqueName: \"kubernetes.io/projected/2b1bc531-5c5d-43f7-a061-60ff3c59d148-kube-api-access-ncvgv\") pod \"payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.781849 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.781809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1209f71b-a54a-4c51-b968-017523b38c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.782041 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.781913    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvgv\" (UniqueName: \"kubernetes.io/projected/2b1bc531-5c5d-43f7-a061-60ff3c59d148-kube-api-access-ncvgv\") pod \"payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.782041 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.781968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rbk\" (UniqueName: \"kubernetes.io/projected/1209f71b-a54a-4c51-b968-017523b38c98-kube-api-access-f8rbk\") pod \"payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.782164 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.782067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1bc531-5c5d-43f7-a061-60ff3c59d148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.782735 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.782716    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1bc531-5c5d-43f7-a061-60ff3c59d148-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.792374 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.792319    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvgv\" (UniqueName: \"kubernetes.io/projected/2b1bc531-5c5d-43f7-a061-60ff3c59d148-kube-api-access-ncvgv\") pod \"payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.858939 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.858904    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:12.883283 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.883255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rbk\" (UniqueName: \"kubernetes.io/projected/1209f71b-a54a-4c51-b968-017523b38c98-kube-api-access-f8rbk\") pod \"payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.883433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.883302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1209f71b-a54a-4c51-b968-017523b38c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.884265 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.884239    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1209f71b-a54a-4c51-b968-017523b38c98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.892496 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.892458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rbk\" (UniqueName: \"kubernetes.io/projected/1209f71b-a54a-4c51-b968-017523b38c98-kube-api-access-f8rbk\") pod \"payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:12.911397 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:12.911363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:13.023015 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.022463    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"]
Jul 29 20:22:13.025276 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:22:13.025243    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1bc531_5c5d_43f7_a061_60ff3c59d148.slice/crio-5c8eb8b1b6e666d15b7af9f60e36c010d252467c67782ae7630d472b2e0ed75a WatchSource:0}: Error finding container 5c8eb8b1b6e666d15b7af9f60e36c010d252467c67782ae7630d472b2e0ed75a: Status 404 returned error can't find the container with id 5c8eb8b1b6e666d15b7af9f60e36c010d252467c67782ae7630d472b2e0ed75a
Jul 29 20:22:13.085974 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.085947    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"]
Jul 29 20:22:13.089964 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:22:13.089934    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1209f71b_a54a_4c51_b968_017523b38c98.slice/crio-4b2ed7b2550f3483557eca42d255476df4138870a11c73a604ee833869e70b74 WatchSource:0}: Error finding container 4b2ed7b2550f3483557eca42d255476df4138870a11c73a604ee833869e70b74: Status 404 returned error can't find the container with id 4b2ed7b2550f3483557eca42d255476df4138870a11c73a604ee833869e70b74
Jul 29 20:22:13.550312 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.550267    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9" event={"ID":"2b1bc531-5c5d-43f7-a061-60ff3c59d148","Type":"ContainerStarted","Data":"00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2"}
Jul 29 20:22:13.550516 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.550338    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9" event={"ID":"2b1bc531-5c5d-43f7-a061-60ff3c59d148","Type":"ContainerStarted","Data":"5c8eb8b1b6e666d15b7af9f60e36c010d252467c67782ae7630d472b2e0ed75a"}
Jul 29 20:22:13.550516 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.550480    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:13.551891 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.551866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd" event={"ID":"1209f71b-a54a-4c51-b968-017523b38c98","Type":"ContainerStarted","Data":"97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c"}
Jul 29 20:22:13.552003 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.551909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd" event={"ID":"1209f71b-a54a-4c51-b968-017523b38c98","Type":"ContainerStarted","Data":"4b2ed7b2550f3483557eca42d255476df4138870a11c73a604ee833869e70b74"}
Jul 29 20:22:13.552003 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.551929    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:13.568256 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.568211    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9" podStartSLOduration=1.5681971799999999 podStartE2EDuration="1.56819718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:22:13.566883612 +0000 UTC m=+3139.525378532" watchObservedRunningTime="2026-07-29 20:22:13.56819718 +0000 UTC m=+3139.526692074"
Jul 29 20:22:13.583172 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.583129    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd" podStartSLOduration=1.5831155749999999 podStartE2EDuration="1.583115575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:22:13.581612774 +0000 UTC m=+3139.540107690" watchObservedRunningTime="2026-07-29 20:22:13.583115575 +0000 UTC m=+3139.541610469"
Jul 29 20:22:13.591188 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.591161    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:13.593692 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.593667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") pod \"maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:13.602283 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.602263    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:13.745877 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.745835    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"]
Jul 29 20:22:13.747412 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:22:13.747387    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc787f1d8_3e11_4b94_8a0a_ab3b3ce6b493.slice/crio-e9f2405857c430311ae29be22c2e7fe7002ea605a415e83b5c29af860c364caf WatchSource:0}: Error finding container e9f2405857c430311ae29be22c2e7fe7002ea605a415e83b5c29af860c364caf: Status 404 returned error can't find the container with id e9f2405857c430311ae29be22c2e7fe7002ea605a415e83b5c29af860c364caf
Jul 29 20:22:13.749167 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:13.749146    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:22:14.560030 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:14.559981    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" event={"ID":"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493","Type":"ContainerStarted","Data":"11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b"}
Jul 29 20:22:14.560436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:14.560113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:14.560436 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:14.560158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" event={"ID":"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493","Type":"ContainerStarted","Data":"e9f2405857c430311ae29be22c2e7fe7002ea605a415e83b5c29af860c364caf"}
Jul 29 20:22:14.575710 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:14.575660    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" podStartSLOduration=3.222123014 podStartE2EDuration="3.575647s" totalImagesPullingTime="353.523986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:22:14.574894056 +0000 UTC m=+3140.533388971" watchObservedRunningTime="2026-07-29 20:22:14.575647 +0000 UTC m=+3140.534141894"
Jul 29 20:22:20.571018 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:20.570941    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:22:24.562051 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:24.562022    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:22:24.562422 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:24.562098    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:22:43.499895 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:43.499860    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hwgbd_9d222932-eac3-4e73-954b-af06b118b1d5/payload-processing/0.log"
Jul 29 20:22:43.621716 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:22:43.621669    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9_2b1bc531-5c5d-43f7-a061-60ff3c59d148/payload-processing/0.log"
Jul 29 20:23:04.961068 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.961034    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"]
Jul 29 20:23:04.964822 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.964799    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:04.967066 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.967044    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-658cad\"/\"openshift-service-ca.crt\""
Jul 29 20:23:04.967183 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.967044    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-658cad\"/\"kube-root-ca.crt\""
Jul 29 20:23:04.967183 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.967152    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-658cad\"/\"ipp-route-658cad-kserve-self-signed-certs\""
Jul 29 20:23:04.967633 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.967616    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-658cad\"/\"default-dockercfg-t9kh5\""
Jul 29 20:23:04.973571 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:04.973544    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"]
Jul 29 20:23:05.084151 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.084109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjggq\" (UniqueName: \"kubernetes.io/projected/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-kube-api-access-vjggq\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.084342 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.084163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tmp-dir\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.084342 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.084227    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-dshm\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.084342 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.084277    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-home\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.084342 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.084323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-model-cache\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.084531 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.084373    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tls-certs\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185381 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185343    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjggq\" (UniqueName: \"kubernetes.io/projected/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-kube-api-access-vjggq\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185390    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tmp-dir\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185452    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-dshm\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-home\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-model-cache\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185539    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tls-certs\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185958 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185887    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tmp-dir\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.185958 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185915    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-model-cache\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.186046 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.185964    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-home\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.187870 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.187847    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-dshm\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.188273 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.188252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tls-certs\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.194135 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.194111    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjggq\" (UniqueName: \"kubernetes.io/projected/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-kube-api-access-vjggq\") pod \"ipp-route-658cad-kserve-6444bbd875-cv47x\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") " pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.275762 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.275694    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:05.424896 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.424865    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"]
Jul 29 20:23:05.426906 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:23:05.426877    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89ee90f5_9d95_4a99_afa2_06bd55bb5ff8.slice/crio-b368ded940bec430d7aabf50bb571380eef908363fe44cf0580c494269b99c3e WatchSource:0}: Error finding container b368ded940bec430d7aabf50bb571380eef908363fe44cf0580c494269b99c3e: Status 404 returned error can't find the container with id b368ded940bec430d7aabf50bb571380eef908363fe44cf0580c494269b99c3e
Jul 29 20:23:05.829431 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.829390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x" event={"ID":"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8","Type":"ContainerStarted","Data":"cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0"}
Jul 29 20:23:05.829431 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.829433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x" event={"ID":"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8","Type":"ContainerStarted","Data":"b368ded940bec430d7aabf50bb571380eef908363fe44cf0580c494269b99c3e"}
Jul 29 20:23:05.846710 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:05.846656    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x" podStartSLOduration=1.846639014 podStartE2EDuration="1.846639014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:23:05.845423749 +0000 UTC m=+3191.803918644" watchObservedRunningTime="2026-07-29 20:23:05.846639014 +0000 UTC m=+3191.805133938"
Jul 29 20:23:15.275976 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:15.275935    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:15.276491 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:15.275995    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:15.288714 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:15.288685    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:15.896075 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:15.896039    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:23:46.026714 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:46.026643    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hwgbd_9d222932-eac3-4e73-954b-af06b118b1d5/payload-processing/0.log"
Jul 29 20:23:46.152291 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:23:46.152251    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9_2b1bc531-5c5d-43f7-a061-60ff3c59d148/payload-processing/0.log"
Jul 29 20:24:46.440898 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.440862    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"]
Jul 29 20:24:46.441797 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.441740    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc" podUID="5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" containerName="payload-pre-processing" containerID="cri-o://341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c" gracePeriod=30
Jul 29 20:24:46.718215 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.718191    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:24:46.881027 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.880994    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqfk9\" (UniqueName: \"kubernetes.io/projected/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-kube-api-access-rqfk9\") pod \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") "
Jul 29 20:24:46.881225 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.881087    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-plugins-config-volume\") pod \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\" (UID: \"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b\") "
Jul 29 20:24:46.881462 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.881432    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" (UID: "5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:24:46.883290 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.883259    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-kube-api-access-rqfk9" (OuterVolumeSpecName: "kube-api-access-rqfk9") pod "5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" (UID: "5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b"). InnerVolumeSpecName "kube-api-access-rqfk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:24:46.981929 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.981862    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfk9\" (UniqueName: \"kubernetes.io/projected/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-kube-api-access-rqfk9\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:24:46.981929 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:46.981888    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:24:47.354723 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.354634    2555 generic.go:356] "Generic (PLEG): container finished" podID="5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" containerID="341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c" exitCode=0
Jul 29 20:24:47.354723 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.354680    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"
Jul 29 20:24:47.354913 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.354728    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc" event={"ID":"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b","Type":"ContainerDied","Data":"341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c"}
Jul 29 20:24:47.354913 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.354784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc" event={"ID":"5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b","Type":"ContainerDied","Data":"778bdedd11b17ec76a91e1715047c1a18caaac2437baa1c9503ddc212b7aa582"}
Jul 29 20:24:47.354913 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.354808    2555 scope.go:117] "RemoveContainer" containerID="341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c"
Jul 29 20:24:47.365433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.365412    2555 scope.go:117] "RemoveContainer" containerID="341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c"
Jul 29 20:24:47.365777 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:24:47.365751    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c\": container with ID starting with 341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c not found: ID does not exist" containerID="341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c"
Jul 29 20:24:47.365869 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.365788    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c"} err="failed to get container status \"341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c\": rpc error: code = NotFound desc = could not find container \"341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c\": container with ID starting with 341abd04d8995319c59b2468fe94e48ab2a13e50a1ab4a4f83d362476987f68c not found: ID does not exist"
Jul 29 20:24:47.375645 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.375618    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"]
Jul 29 20:24:47.381733 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:47.381713    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-658cad-7fb5d8d54d-4vffc"]
Jul 29 20:24:48.659833 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:48.659801    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" path="/var/lib/kubelet/pods/5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b/volumes"
Jul 29 20:24:51.935944 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:51.935909    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"]
Jul 29 20:24:54.782570 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:54.782530    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:24:54.788143 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:24:54.788122    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rg625_fc12525f-9d1b-4f9b-b066-4781885c641d/ovn-acl-logging/0.log"
Jul 29 20:25:06.956785 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:06.956738    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x" podUID="89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" containerName="main" containerID="cri-o://cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0" gracePeriod=45
Jul 29 20:25:07.214848 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.214785    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:25:07.274289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274249    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjggq\" (UniqueName: \"kubernetes.io/projected/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-kube-api-access-vjggq\") pod \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") "
Jul 29 20:25:07.274458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tmp-dir\") pod \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") "
Jul 29 20:25:07.274458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274420    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tls-certs\") pod \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") "
Jul 29 20:25:07.274458 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274445    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-dshm\") pod \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") "
Jul 29 20:25:07.274624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274479    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-home\") pod \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") "
Jul 29 20:25:07.274624 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274506    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-model-cache\") pod \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\" (UID: \"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8\") "
Jul 29 20:25:07.274753 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274681    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" (UID: "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:25:07.274809 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274769    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-home" (OuterVolumeSpecName: "home") pod "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" (UID: "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:25:07.274934 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274914    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-model-cache" (OuterVolumeSpecName: "model-cache") pod "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" (UID: "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:25:07.274995 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274924    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tmp-dir\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:25:07.274995 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.274954    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-home\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:25:07.276958 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.276927    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-dshm" (OuterVolumeSpecName: "dshm") pod "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" (UID: "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:25:07.277213 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.277188    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" (UID: "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:25:07.277213 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.277197    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-kube-api-access-vjggq" (OuterVolumeSpecName: "kube-api-access-vjggq") pod "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" (UID: "89ee90f5-9d95-4a99-afa2-06bd55bb5ff8"). InnerVolumeSpecName "kube-api-access-vjggq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:25:07.376127 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.376088    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-tls-certs\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:25:07.376127 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.376120    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-dshm\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:25:07.376127 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.376129    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-model-cache\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:25:07.376127 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.376140    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjggq\" (UniqueName: \"kubernetes.io/projected/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8-kube-api-access-vjggq\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:25:07.463231 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.463194    2555 generic.go:356] "Generic (PLEG): container finished" podID="89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" containerID="cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0" exitCode=0
Jul 29 20:25:07.463231 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.463234    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"
Jul 29 20:25:07.463456 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.463292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x" event={"ID":"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8","Type":"ContainerDied","Data":"cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0"}
Jul 29 20:25:07.463456 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.463333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x" event={"ID":"89ee90f5-9d95-4a99-afa2-06bd55bb5ff8","Type":"ContainerDied","Data":"b368ded940bec430d7aabf50bb571380eef908363fe44cf0580c494269b99c3e"}
Jul 29 20:25:07.463456 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.463351    2555 scope.go:117] "RemoveContainer" containerID="cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0"
Jul 29 20:25:07.472317 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.472299    2555 scope.go:117] "RemoveContainer" containerID="cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0"
Jul 29 20:25:07.472601 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:25:07.472556    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0\": container with ID starting with cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0 not found: ID does not exist" containerID="cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0"
Jul 29 20:25:07.472682 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.472601    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0"} err="failed to get container status \"cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0\": rpc error: code = NotFound desc = could not find container \"cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0\": container with ID starting with cbf09ea79427cf9f86dd28ea8e95c9fd84f5404cf1a3f5b7c092e6afdcd535c0 not found: ID does not exist"
Jul 29 20:25:07.483746 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.483719    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"]
Jul 29 20:25:07.488533 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:07.488511    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-658cad/ipp-route-658cad-kserve-6444bbd875-cv47x"]
Jul 29 20:25:08.659234 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:25:08.659199    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" path="/var/lib/kubelet/pods/89ee90f5-9d95-4a99-afa2-06bd55bb5ff8/volumes"
Jul 29 20:26:00.713553 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:00.713519    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"]
Jul 29 20:26:00.714074 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:00.713798    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" podUID="c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" containerName="maas-api" containerID="cri-o://11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b" gracePeriod=30
Jul 29 20:26:00.767356 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:00.767324    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"]
Jul 29 20:26:00.767611 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:00.767548    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9" podUID="2b1bc531-5c5d-43f7-a061-60ff3c59d148" containerName="payload-processing" containerID="cri-o://00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2" gracePeriod=30
Jul 29 20:26:00.776045 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:00.775937    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"]
Jul 29 20:26:00.776188 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:00.776148    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd" podUID="1209f71b-a54a-4c51-b968-017523b38c98" containerName="payload-pre-processing" containerID="cri-o://97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c" gracePeriod=30
Jul 29 20:26:01.094057 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.094032    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:26:01.099678 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.099655    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:26:01.118200 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.118176    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:26:01.182095 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.182064    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8rbk\" (UniqueName: \"kubernetes.io/projected/1209f71b-a54a-4c51-b968-017523b38c98-kube-api-access-f8rbk\") pod \"1209f71b-a54a-4c51-b968-017523b38c98\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") "
Jul 29 20:26:01.182277 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.182155    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1209f71b-a54a-4c51-b968-017523b38c98-plugins-config-volume\") pod \"1209f71b-a54a-4c51-b968-017523b38c98\" (UID: \"1209f71b-a54a-4c51-b968-017523b38c98\") "
Jul 29 20:26:01.182277 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.182222    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvgv\" (UniqueName: \"kubernetes.io/projected/2b1bc531-5c5d-43f7-a061-60ff3c59d148-kube-api-access-ncvgv\") pod \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") "
Jul 29 20:26:01.182277 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.182244    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1bc531-5c5d-43f7-a061-60ff3c59d148-plugins-config-volume\") pod \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\" (UID: \"2b1bc531-5c5d-43f7-a061-60ff3c59d148\") "
Jul 29 20:26:01.182696 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.182650    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1209f71b-a54a-4c51-b968-017523b38c98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1209f71b-a54a-4c51-b968-017523b38c98" (UID: "1209f71b-a54a-4c51-b968-017523b38c98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:01.183005 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.182749    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1bc531-5c5d-43f7-a061-60ff3c59d148-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b1bc531-5c5d-43f7-a061-60ff3c59d148" (UID: "2b1bc531-5c5d-43f7-a061-60ff3c59d148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:01.184485 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.184458    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1209f71b-a54a-4c51-b968-017523b38c98-kube-api-access-f8rbk" (OuterVolumeSpecName: "kube-api-access-f8rbk") pod "1209f71b-a54a-4c51-b968-017523b38c98" (UID: "1209f71b-a54a-4c51-b968-017523b38c98"). InnerVolumeSpecName "kube-api-access-f8rbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:01.184634 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.184609    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1bc531-5c5d-43f7-a061-60ff3c59d148-kube-api-access-ncvgv" (OuterVolumeSpecName: "kube-api-access-ncvgv") pod "2b1bc531-5c5d-43f7-a061-60ff3c59d148" (UID: "2b1bc531-5c5d-43f7-a061-60ff3c59d148"). InnerVolumeSpecName "kube-api-access-ncvgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:01.283235 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283153    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") pod \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") "
Jul 29 20:26:01.283382 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283303    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfsf9\" (UniqueName: \"kubernetes.io/projected/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-kube-api-access-xfsf9\") pod \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") "
Jul 29 20:26:01.283382 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283369    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-openshift-service-ca\") pod \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\" (UID: \"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493\") "
Jul 29 20:26:01.283701 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283683    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8rbk\" (UniqueName: \"kubernetes.io/projected/1209f71b-a54a-4c51-b968-017523b38c98-kube-api-access-f8rbk\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.283768 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283705    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1209f71b-a54a-4c51-b968-017523b38c98-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.283768 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283716    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvgv\" (UniqueName: \"kubernetes.io/projected/2b1bc531-5c5d-43f7-a061-60ff3c59d148-kube-api-access-ncvgv\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.283768 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283725    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1bc531-5c5d-43f7-a061-60ff3c59d148-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.283768 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.283727    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" (UID: "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:01.285586 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.285554    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" (UID: "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:26:01.285639 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.285597    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-kube-api-access-xfsf9" (OuterVolumeSpecName: "kube-api-access-xfsf9") pod "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" (UID: "c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493"). InnerVolumeSpecName "kube-api-access-xfsf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:01.384932 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.384889    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfsf9\" (UniqueName: \"kubernetes.io/projected/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-kube-api-access-xfsf9\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.384932 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.384922    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.384932 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.384934    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:01.729267 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.729231    2555 generic.go:356] "Generic (PLEG): container finished" podID="1209f71b-a54a-4c51-b968-017523b38c98" containerID="97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c" exitCode=0
Jul 29 20:26:01.729776 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.729278    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"
Jul 29 20:26:01.729776 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.729317    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd" event={"ID":"1209f71b-a54a-4c51-b968-017523b38c98","Type":"ContainerDied","Data":"97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c"}
Jul 29 20:26:01.729776 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.729375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd" event={"ID":"1209f71b-a54a-4c51-b968-017523b38c98","Type":"ContainerDied","Data":"4b2ed7b2550f3483557eca42d255476df4138870a11c73a604ee833869e70b74"}
Jul 29 20:26:01.729776 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.729398    2555 scope.go:117] "RemoveContainer" containerID="97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c"
Jul 29 20:26:01.731113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.730738    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b1bc531-5c5d-43f7-a061-60ff3c59d148" containerID="00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2" exitCode=0
Jul 29 20:26:01.731113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.730779    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"
Jul 29 20:26:01.731113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.730832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9" event={"ID":"2b1bc531-5c5d-43f7-a061-60ff3c59d148","Type":"ContainerDied","Data":"00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2"}
Jul 29 20:26:01.731113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.730892    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9" event={"ID":"2b1bc531-5c5d-43f7-a061-60ff3c59d148","Type":"ContainerDied","Data":"5c8eb8b1b6e666d15b7af9f60e36c010d252467c67782ae7630d472b2e0ed75a"}
Jul 29 20:26:01.734291 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.733063    2555 generic.go:356] "Generic (PLEG): container finished" podID="c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" containerID="11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b" exitCode=0
Jul 29 20:26:01.734291 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.733128    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" event={"ID":"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493","Type":"ContainerDied","Data":"11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b"}
Jul 29 20:26:01.734291 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.733169    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488" event={"ID":"c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493","Type":"ContainerDied","Data":"e9f2405857c430311ae29be22c2e7fe7002ea605a415e83b5c29af860c364caf"}
Jul 29 20:26:01.734291 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.733354    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"
Jul 29 20:26:01.746267 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.746245    2555 scope.go:117] "RemoveContainer" containerID="97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c"
Jul 29 20:26:01.746660 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:01.746628    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c\": container with ID starting with 97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c not found: ID does not exist" containerID="97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c"
Jul 29 20:26:01.746761 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.746675    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c"} err="failed to get container status \"97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c\": rpc error: code = NotFound desc = could not find container \"97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c\": container with ID starting with 97091c251fc96c13b841473b8d326fb3f4b1a3bf7ccf26544b4ef54182067c4c not found: ID does not exist"
Jul 29 20:26:01.746761 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.746701    2555 scope.go:117] "RemoveContainer" containerID="00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2"
Jul 29 20:26:01.754886 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.754869    2555 scope.go:117] "RemoveContainer" containerID="00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2"
Jul 29 20:26:01.755132 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:01.755113    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2\": container with ID starting with 00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2 not found: ID does not exist" containerID="00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2"
Jul 29 20:26:01.755183 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.755138    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2"} err="failed to get container status \"00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2\": rpc error: code = NotFound desc = could not find container \"00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2\": container with ID starting with 00e0966bc0a51ae1dcda85a636d77ca9475fc4ee5e8e2c55cec598bccdf58cc2 not found: ID does not exist"
Jul 29 20:26:01.755183 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.755154    2555 scope.go:117] "RemoveContainer" containerID="11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b"
Jul 29 20:26:01.760481 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.760455    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"]
Jul 29 20:26:01.762558 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.762530    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-259e5b-66c8bd5684-n7tsd"]
Jul 29 20:26:01.763509 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.763490    2555 scope.go:117] "RemoveContainer" containerID="11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b"
Jul 29 20:26:01.763840 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:01.763822    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b\": container with ID starting with 11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b not found: ID does not exist" containerID="11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b"
Jul 29 20:26:01.763905 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.763848    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b"} err="failed to get container status \"11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b\": rpc error: code = NotFound desc = could not find container \"11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b\": container with ID starting with 11a3821f1e7ca048f6854cbe2ba846f51e8b1a76852a8ea7ec933aaabd64cf9b not found: ID does not exist"
Jul 29 20:26:01.774990 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.774967    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"]
Jul 29 20:26:01.779368 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.779348    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-259e5b-5595d665dd-jwjv9"]
Jul 29 20:26:01.794081 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.794056    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"]
Jul 29 20:26:01.798269 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:01.798246    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-259e5b-ffbcfb89d-qw488"]
Jul 29 20:26:02.660433 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:02.660396    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1209f71b-a54a-4c51-b968-017523b38c98" path="/var/lib/kubelet/pods/1209f71b-a54a-4c51-b968-017523b38c98/volumes"
Jul 29 20:26:02.660908 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:02.660885    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1bc531-5c5d-43f7-a061-60ff3c59d148" path="/var/lib/kubelet/pods/2b1bc531-5c5d-43f7-a061-60ff3c59d148/volumes"
Jul 29 20:26:02.661341 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:02.661320    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" path="/var/lib/kubelet/pods/c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493/volumes"
Jul 29 20:26:18.899315 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:18.899279    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"]
Jul 29 20:26:18.899849 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:18.899619    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" podUID="8013b929-27b9-4574-a68b-2d2c4f59a10e" containerName="maas-api" containerID="cri-o://fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed" gracePeriod=30
Jul 29 20:26:18.968038 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:18.968002    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"]
Jul 29 20:26:18.968331 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:18.968268    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77" podUID="83a059d6-879c-411e-a317-63d7185089a9" containerName="payload-processing" containerID="cri-o://e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24" gracePeriod=30
Jul 29 20:26:18.975875 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:18.975840    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"]
Jul 29 20:26:18.976101 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:18.976067    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz" podUID="5c49992e-de73-4a27-9bac-59f679ac5b9c" containerName="payload-pre-processing" containerID="cri-o://e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8" gracePeriod=30
Jul 29 20:26:19.265233 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.265207    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:26:19.268130 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.268107    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:26:19.271505 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.271485    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:26:19.359094 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359055    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls\") pod \"8013b929-27b9-4574-a68b-2d2c4f59a10e\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") "
Jul 29 20:26:19.359264 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359139    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c49992e-de73-4a27-9bac-59f679ac5b9c-plugins-config-volume\") pod \"5c49992e-de73-4a27-9bac-59f679ac5b9c\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") "
Jul 29 20:26:19.359264 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359166    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a059d6-879c-411e-a317-63d7185089a9-plugins-config-volume\") pod \"83a059d6-879c-411e-a317-63d7185089a9\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") "
Jul 29 20:26:19.359264 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359192    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94vsw\" (UniqueName: \"kubernetes.io/projected/8013b929-27b9-4574-a68b-2d2c4f59a10e-kube-api-access-94vsw\") pod \"8013b929-27b9-4574-a68b-2d2c4f59a10e\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") "
Jul 29 20:26:19.359264 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359219    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qshwz\" (UniqueName: \"kubernetes.io/projected/83a059d6-879c-411e-a317-63d7185089a9-kube-api-access-qshwz\") pod \"83a059d6-879c-411e-a317-63d7185089a9\" (UID: \"83a059d6-879c-411e-a317-63d7185089a9\") "
Jul 29 20:26:19.359264 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359253    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8013b929-27b9-4574-a68b-2d2c4f59a10e-openshift-service-ca\") pod \"8013b929-27b9-4574-a68b-2d2c4f59a10e\" (UID: \"8013b929-27b9-4574-a68b-2d2c4f59a10e\") "
Jul 29 20:26:19.359539 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359304    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26xxw\" (UniqueName: \"kubernetes.io/projected/5c49992e-de73-4a27-9bac-59f679ac5b9c-kube-api-access-26xxw\") pod \"5c49992e-de73-4a27-9bac-59f679ac5b9c\" (UID: \"5c49992e-de73-4a27-9bac-59f679ac5b9c\") "
Jul 29 20:26:19.359684 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359638    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a059d6-879c-411e-a317-63d7185089a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83a059d6-879c-411e-a317-63d7185089a9" (UID: "83a059d6-879c-411e-a317-63d7185089a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:19.359684 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359657    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c49992e-de73-4a27-9bac-59f679ac5b9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c49992e-de73-4a27-9bac-59f679ac5b9c" (UID: "5c49992e-de73-4a27-9bac-59f679ac5b9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:19.360049 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.359694    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8013b929-27b9-4574-a68b-2d2c4f59a10e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8013b929-27b9-4574-a68b-2d2c4f59a10e" (UID: "8013b929-27b9-4574-a68b-2d2c4f59a10e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:19.361407 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.361381    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8013b929-27b9-4574-a68b-2d2c4f59a10e" (UID: "8013b929-27b9-4574-a68b-2d2c4f59a10e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:26:19.361504 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.361434    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8013b929-27b9-4574-a68b-2d2c4f59a10e-kube-api-access-94vsw" (OuterVolumeSpecName: "kube-api-access-94vsw") pod "8013b929-27b9-4574-a68b-2d2c4f59a10e" (UID: "8013b929-27b9-4574-a68b-2d2c4f59a10e"). InnerVolumeSpecName "kube-api-access-94vsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:19.361504 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.361476    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a059d6-879c-411e-a317-63d7185089a9-kube-api-access-qshwz" (OuterVolumeSpecName: "kube-api-access-qshwz") pod "83a059d6-879c-411e-a317-63d7185089a9" (UID: "83a059d6-879c-411e-a317-63d7185089a9"). InnerVolumeSpecName "kube-api-access-qshwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:19.361850 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.361828    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c49992e-de73-4a27-9bac-59f679ac5b9c-kube-api-access-26xxw" (OuterVolumeSpecName: "kube-api-access-26xxw") pod "5c49992e-de73-4a27-9bac-59f679ac5b9c" (UID: "5c49992e-de73-4a27-9bac-59f679ac5b9c"). InnerVolumeSpecName "kube-api-access-26xxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460413    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26xxw\" (UniqueName: \"kubernetes.io/projected/5c49992e-de73-4a27-9bac-59f679ac5b9c-kube-api-access-26xxw\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460438    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8013b929-27b9-4574-a68b-2d2c4f59a10e-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460448    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c49992e-de73-4a27-9bac-59f679ac5b9c-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460458    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a059d6-879c-411e-a317-63d7185089a9-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460468    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94vsw\" (UniqueName: \"kubernetes.io/projected/8013b929-27b9-4574-a68b-2d2c4f59a10e-kube-api-access-94vsw\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460476    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshwz\" (UniqueName: \"kubernetes.io/projected/83a059d6-879c-411e-a317-63d7185089a9-kube-api-access-qshwz\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.460486 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.460484    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8013b929-27b9-4574-a68b-2d2c4f59a10e-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:19.830082 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.829998    2555 generic.go:356] "Generic (PLEG): container finished" podID="5c49992e-de73-4a27-9bac-59f679ac5b9c" containerID="e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8" exitCode=0
Jul 29 20:26:19.830082 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.830060    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"
Jul 29 20:26:19.830308 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.830092    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz" event={"ID":"5c49992e-de73-4a27-9bac-59f679ac5b9c","Type":"ContainerDied","Data":"e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8"}
Jul 29 20:26:19.830308 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.830158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz" event={"ID":"5c49992e-de73-4a27-9bac-59f679ac5b9c","Type":"ContainerDied","Data":"42d100b7999c992375b8b462ffd3617e9be76069ab37fcda0b550ee11274d149"}
Jul 29 20:26:19.830308 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.830193    2555 scope.go:117] "RemoveContainer" containerID="e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8"
Jul 29 20:26:19.831409 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.831382    2555 generic.go:356] "Generic (PLEG): container finished" podID="8013b929-27b9-4574-a68b-2d2c4f59a10e" containerID="fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed" exitCode=0
Jul 29 20:26:19.831524 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.831436    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"
Jul 29 20:26:19.831524 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.831466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" event={"ID":"8013b929-27b9-4574-a68b-2d2c4f59a10e","Type":"ContainerDied","Data":"fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed"}
Jul 29 20:26:19.831524 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.831514    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t" event={"ID":"8013b929-27b9-4574-a68b-2d2c4f59a10e","Type":"ContainerDied","Data":"96ba2a88d5251ef1e8d9d32ff1a9a78495e58ed18ded84a7d5b50dc44ecda8d5"}
Jul 29 20:26:19.833194 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.833014    2555 generic.go:356] "Generic (PLEG): container finished" podID="83a059d6-879c-411e-a317-63d7185089a9" containerID="e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24" exitCode=0
Jul 29 20:26:19.833194 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.833062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77" event={"ID":"83a059d6-879c-411e-a317-63d7185089a9","Type":"ContainerDied","Data":"e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24"}
Jul 29 20:26:19.833194 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.833077    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"
Jul 29 20:26:19.833194 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.833106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77" event={"ID":"83a059d6-879c-411e-a317-63d7185089a9","Type":"ContainerDied","Data":"39aa858494730a1d4a039992cac3b0a45e17b0b7449ab9fca62596393c8e66fa"}
Jul 29 20:26:19.841447 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.841212    2555 scope.go:117] "RemoveContainer" containerID="e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8"
Jul 29 20:26:19.841662 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:19.841572    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8\": container with ID starting with e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8 not found: ID does not exist" containerID="e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8"
Jul 29 20:26:19.841757 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.841675    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8"} err="failed to get container status \"e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8\": rpc error: code = NotFound desc = could not find container \"e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8\": container with ID starting with e9a56f3530512216579f97a459a24f2b4a6263579e7ec693d56a2fa9d5e8f3d8 not found: ID does not exist"
Jul 29 20:26:19.841757 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.841698    2555 scope.go:117] "RemoveContainer" containerID="fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed"
Jul 29 20:26:19.850994 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.850974    2555 scope.go:117] "RemoveContainer" containerID="fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed"
Jul 29 20:26:19.851259 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:19.851238    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed\": container with ID starting with fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed not found: ID does not exist" containerID="fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed"
Jul 29 20:26:19.851294 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.851268    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed"} err="failed to get container status \"fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed\": rpc error: code = NotFound desc = could not find container \"fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed\": container with ID starting with fb7322c3cc1baec576ddf3c2dd90ce2e098bd756af0c1b609b36f2effe6163ed not found: ID does not exist"
Jul 29 20:26:19.851294 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.851285    2555 scope.go:117] "RemoveContainer" containerID="e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24"
Jul 29 20:26:19.854869 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.854848    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"]
Jul 29 20:26:19.858995 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.858978    2555 scope.go:117] "RemoveContainer" containerID="e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24"
Jul 29 20:26:19.859447 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:19.859363    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24\": container with ID starting with e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24 not found: ID does not exist" containerID="e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24"
Jul 29 20:26:19.859447 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.859413    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24"} err="failed to get container status \"e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24\": rpc error: code = NotFound desc = could not find container \"e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24\": container with ID starting with e5ca6089d85e4c8e6d6c5ac99c898fafd04524165002326444f4183446c61d24 not found: ID does not exist"
Jul 29 20:26:19.861256 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.861235    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-7d4a80-5fb4fcf564-lfkbz"]
Jul 29 20:26:19.871277 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.871257    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"]
Jul 29 20:26:19.873723 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.873705    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-7d4a80-b98cd775d-c7k77"]
Jul 29 20:26:19.884543 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.884516    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"]
Jul 29 20:26:19.888283 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:19.888256    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7d4a80-5fcc749c49-6nq5t"]
Jul 29 20:26:20.659023 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:20.658985    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c49992e-de73-4a27-9bac-59f679ac5b9c" path="/var/lib/kubelet/pods/5c49992e-de73-4a27-9bac-59f679ac5b9c/volumes"
Jul 29 20:26:20.659568 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:20.659548    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8013b929-27b9-4574-a68b-2d2c4f59a10e" path="/var/lib/kubelet/pods/8013b929-27b9-4574-a68b-2d2c4f59a10e/volumes"
Jul 29 20:26:20.659952 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:20.659933    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a059d6-879c-411e-a317-63d7185089a9" path="/var/lib/kubelet/pods/83a059d6-879c-411e-a317-63d7185089a9/volumes"
Jul 29 20:26:36.962053 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:36.962015    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"]
Jul 29 20:26:36.962609 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:36.962281    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5" podUID="38d182f2-dbfa-4186-9377-69ba6d0c27aa" containerName="maas-api" containerID="cri-o://903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f" gracePeriod=30
Jul 29 20:26:37.026763 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.026729    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"]
Jul 29 20:26:37.026988 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.026965    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67" podUID="53336460-7a6b-4737-84f6-06248ed5eb4c" containerName="payload-processing" containerID="cri-o://2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a" gracePeriod=30
Jul 29 20:26:37.281096 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.281070    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:26:37.304095 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.304072    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:26:37.424190 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424155    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d182f2-dbfa-4186-9377-69ba6d0c27aa-maas-api-tls\") pod \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") "
Jul 29 20:26:37.424379 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424210    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsc8w\" (UniqueName: \"kubernetes.io/projected/38d182f2-dbfa-4186-9377-69ba6d0c27aa-kube-api-access-nsc8w\") pod \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") "
Jul 29 20:26:37.424379 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424245    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53336460-7a6b-4737-84f6-06248ed5eb4c-plugins-config-volume\") pod \"53336460-7a6b-4737-84f6-06248ed5eb4c\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") "
Jul 29 20:26:37.424379 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424310    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d182f2-dbfa-4186-9377-69ba6d0c27aa-openshift-service-ca\") pod \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\" (UID: \"38d182f2-dbfa-4186-9377-69ba6d0c27aa\") "
Jul 29 20:26:37.424379 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424372    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b25r\" (UniqueName: \"kubernetes.io/projected/53336460-7a6b-4737-84f6-06248ed5eb4c-kube-api-access-8b25r\") pod \"53336460-7a6b-4737-84f6-06248ed5eb4c\" (UID: \"53336460-7a6b-4737-84f6-06248ed5eb4c\") "
Jul 29 20:26:37.424711 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424677    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53336460-7a6b-4737-84f6-06248ed5eb4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53336460-7a6b-4737-84f6-06248ed5eb4c" (UID: "53336460-7a6b-4737-84f6-06248ed5eb4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:37.424793 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.424766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d182f2-dbfa-4186-9377-69ba6d0c27aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38d182f2-dbfa-4186-9377-69ba6d0c27aa" (UID: "38d182f2-dbfa-4186-9377-69ba6d0c27aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:26:37.426700 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.426675    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d182f2-dbfa-4186-9377-69ba6d0c27aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38d182f2-dbfa-4186-9377-69ba6d0c27aa" (UID: "38d182f2-dbfa-4186-9377-69ba6d0c27aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:26:37.426802 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.426727    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d182f2-dbfa-4186-9377-69ba6d0c27aa-kube-api-access-nsc8w" (OuterVolumeSpecName: "kube-api-access-nsc8w") pod "38d182f2-dbfa-4186-9377-69ba6d0c27aa" (UID: "38d182f2-dbfa-4186-9377-69ba6d0c27aa"). InnerVolumeSpecName "kube-api-access-nsc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:37.426802 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.426755    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53336460-7a6b-4737-84f6-06248ed5eb4c-kube-api-access-8b25r" (OuterVolumeSpecName: "kube-api-access-8b25r") pod "53336460-7a6b-4737-84f6-06248ed5eb4c" (UID: "53336460-7a6b-4737-84f6-06248ed5eb4c"). InnerVolumeSpecName "kube-api-access-8b25r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:26:37.525300 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.525212    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d182f2-dbfa-4186-9377-69ba6d0c27aa-maas-api-tls\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:37.525300 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.525244    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsc8w\" (UniqueName: \"kubernetes.io/projected/38d182f2-dbfa-4186-9377-69ba6d0c27aa-kube-api-access-nsc8w\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:37.525300 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.525254    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53336460-7a6b-4737-84f6-06248ed5eb4c-plugins-config-volume\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:37.525300 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.525262    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d182f2-dbfa-4186-9377-69ba6d0c27aa-openshift-service-ca\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:37.525300 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.525270    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b25r\" (UniqueName: \"kubernetes.io/projected/53336460-7a6b-4737-84f6-06248ed5eb4c-kube-api-access-8b25r\") on node \"ip-10-0-138-139.ec2.internal\" DevicePath \"\""
Jul 29 20:26:37.927276 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.927193    2555 generic.go:356] "Generic (PLEG): container finished" podID="38d182f2-dbfa-4186-9377-69ba6d0c27aa" containerID="903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f" exitCode=0
Jul 29 20:26:37.927276 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.927259    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"
Jul 29 20:26:37.927480 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.927286    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5" event={"ID":"38d182f2-dbfa-4186-9377-69ba6d0c27aa","Type":"ContainerDied","Data":"903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f"}
Jul 29 20:26:37.927480 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.927345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5" event={"ID":"38d182f2-dbfa-4186-9377-69ba6d0c27aa","Type":"ContainerDied","Data":"b77105d1e6d6b55c247bacd6e94d1f540ab3631351a4051b637719ecda868cef"}
Jul 29 20:26:37.927480 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.927379    2555 scope.go:117] "RemoveContainer" containerID="903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f"
Jul 29 20:26:37.928739 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.928720    2555 generic.go:356] "Generic (PLEG): container finished" podID="53336460-7a6b-4737-84f6-06248ed5eb4c" containerID="2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a" exitCode=0
Jul 29 20:26:37.928833 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.928749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67" event={"ID":"53336460-7a6b-4737-84f6-06248ed5eb4c","Type":"ContainerDied","Data":"2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a"}
Jul 29 20:26:37.928833 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.928778    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"
Jul 29 20:26:37.928833 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.928817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67" event={"ID":"53336460-7a6b-4737-84f6-06248ed5eb4c","Type":"ContainerDied","Data":"84196bbfe2dc85f8c7d90905d3ab3c671329c1033ac0afe0f3e7e030128c08fa"}
Jul 29 20:26:37.936931 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.936911    2555 scope.go:117] "RemoveContainer" containerID="903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f"
Jul 29 20:26:37.937371 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:37.937332    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f\": container with ID starting with 903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f not found: ID does not exist" containerID="903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f"
Jul 29 20:26:37.937464 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.937384    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f"} err="failed to get container status \"903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f\": rpc error: code = NotFound desc = could not find container \"903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f\": container with ID starting with 903bc77fc9380e2dd64adeb7b6c862d0ce8d8abd7caa51c401d97d6a01a0cc6f not found: ID does not exist"
Jul 29 20:26:37.937464 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.937409    2555 scope.go:117] "RemoveContainer" containerID="2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a"
Jul 29 20:26:37.945930 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.945912    2555 scope.go:117] "RemoveContainer" containerID="2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a"
Jul 29 20:26:37.946197 ip-10-0-138-139 kubenswrapper[2555]: E0729 20:26:37.946179    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a\": container with ID starting with 2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a not found: ID does not exist" containerID="2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a"
Jul 29 20:26:37.946249 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.946208    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a"} err="failed to get container status \"2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a\": rpc error: code = NotFound desc = could not find container \"2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a\": container with ID starting with 2463258bbcceb52c0c2cdec23bc9fec0cf7b0fa7280d0429dcf696c8f92e265a not found: ID does not exist"
Jul 29 20:26:37.953484 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.953461    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"]
Jul 29 20:26:37.959992 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.959970    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1a1eaa-8784cf45b-brwt5"]
Jul 29 20:26:37.970606 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.970568    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"]
Jul 29 20:26:37.977205 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:37.977183    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1a1eaa-66c9c9858-5rb67"]
Jul 29 20:26:38.658894 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:38.658862    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d182f2-dbfa-4186-9377-69ba6d0c27aa" path="/var/lib/kubelet/pods/38d182f2-dbfa-4186-9377-69ba6d0c27aa/volumes"
Jul 29 20:26:38.659215 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:38.659202    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53336460-7a6b-4737-84f6-06248ed5eb4c" path="/var/lib/kubelet/pods/53336460-7a6b-4737-84f6-06248ed5eb4c/volumes"
Jul 29 20:26:50.290796 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:50.290712    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c74855f4-dmcx7_6fa5223b-f9d0-41d4-92af-ffa4896de0f9/manager/0.log"
Jul 29 20:26:50.641240 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:50.641164    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d75b7b96-lbglf_684e45cf-b6db-448c-b7ce-73fd08a66308/copy-manifests/0.log"
Jul 29 20:26:50.648344 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:50.648318    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d75b7b96-lbglf_684e45cf-b6db-448c-b7ce-73fd08a66308/manager/0.log"
Jul 29 20:26:51.251215 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:51.251176    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-776f87874c-b7jnb_7055208f-2e5b-4c9f-8a57-ac1133fdaffe/maas-api/0.log"
Jul 29 20:26:53.288189 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:53.288159    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-x29j4_cd7e4b4c-cf34-4858-85f3-190ffe31edc2/kuadrant-console-plugin/0.log"
Jul 29 20:26:53.520052 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:53.520006    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k7n79_7d46e063-fdfa-463f-8b83-690b5a74c4cb/manager/0.log"
Jul 29 20:26:54.296895 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:54.296862    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cc65fd5df-njhfl_c9e686ba-1b46-4989-b6e5-ba3adec565c4/kube-auth-proxy/0.log"
Jul 29 20:26:54.759048 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:54.758984    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hwgbd_9d222932-eac3-4e73-954b-af06b118b1d5/payload-processing/0.log"
Jul 29 20:26:55.514500 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:55.514467    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9zhpn_1d688b1d-dfad-4a6e-a64d-d71e971859c3/main/0.log"
Jul 29 20:26:55.842157 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:26:55.842067    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-d9zt4_45219a0f-5f75-4ca0-8c62-6b4bdd297151/main/0.log"
Jul 29 20:27:03.826900 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:03.826870    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gr9w2_e0d368bf-30b8-4d18-a17f-306eee453ddd/global-pull-secret-syncer/0.log"
Jul 29 20:27:03.956046 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:03.956012    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h27gc_1c633756-10c1-43d4-9dc0-b722ffa6deb0/konnectivity-agent/0.log"
Jul 29 20:27:04.047289 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:04.047250    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-139.ec2.internal_ca69d5b906524dbafb4263c4750a6c45/haproxy/0.log"
Jul 29 20:27:09.839318 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:09.839289    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-x29j4_cd7e4b4c-cf34-4858-85f3-190ffe31edc2/kuadrant-console-plugin/0.log"
Jul 29 20:27:10.141714 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:10.141621    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k7n79_7d46e063-fdfa-463f-8b83-690b5a74c4cb/manager/0.log"
Jul 29 20:27:12.221806 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.221767    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7665554cb6-zf7ln_eeb26ced-3607-457c-88f6-b7373a5206ef/metrics-server/0.log"
Jul 29 20:27:12.283306 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.283274    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lgvh_8e4b965e-52de-4fde-8617-68548bf1b8a1/node-exporter/0.log"
Jul 29 20:27:12.310686 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.310666    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lgvh_8e4b965e-52de-4fde-8617-68548bf1b8a1/kube-rbac-proxy/0.log"
Jul 29 20:27:12.333951 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.333928    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lgvh_8e4b965e-52de-4fde-8617-68548bf1b8a1/init-textfile/0.log"
Jul 29 20:27:12.520504 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.520426    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hv6qd_c35df15c-f6ae-4903-850d-352c4f0d8171/kube-rbac-proxy-main/0.log"
Jul 29 20:27:12.543176 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.543148    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hv6qd_c35df15c-f6ae-4903-850d-352c4f0d8171/kube-rbac-proxy-self/0.log"
Jul 29 20:27:12.566189 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.566165    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-hv6qd_c35df15c-f6ae-4903-850d-352c4f0d8171/openshift-state-metrics/0.log"
Jul 29 20:27:12.615351 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.615325    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/prometheus/0.log"
Jul 29 20:27:12.637251 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.637222    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/config-reloader/0.log"
Jul 29 20:27:12.665807 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.665785    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/thanos-sidecar/0.log"
Jul 29 20:27:12.688307 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.688268    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/kube-rbac-proxy-web/0.log"
Jul 29 20:27:12.711113 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.711087    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/kube-rbac-proxy/0.log"
Jul 29 20:27:12.736427 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.736404    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/kube-rbac-proxy-thanos/0.log"
Jul 29 20:27:12.765915 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.765897    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bcd3fba0-1090-4d47-b41a-97380b98ff69/init-config-reloader/0.log"
Jul 29 20:27:12.802665 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.802566    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-4zlg5_3b6e2627-48d0-4286-a849-2d9c667293cc/prometheus-operator/0.log"
Jul 29 20:27:12.819946 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.819925    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-4zlg5_3b6e2627-48d0-4286-a849-2d9c667293cc/kube-rbac-proxy/0.log"
Jul 29 20:27:12.958813 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.958783    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c757d79-g5bhl_b892161e-6e80-46d2-bb45-64e5aefb98dc/thanos-query/0.log"
Jul 29 20:27:12.982509 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:12.982485    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c757d79-g5bhl_b892161e-6e80-46d2-bb45-64e5aefb98dc/kube-rbac-proxy-web/0.log"
Jul 29 20:27:13.005145 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:13.005119    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c757d79-g5bhl_b892161e-6e80-46d2-bb45-64e5aefb98dc/kube-rbac-proxy/0.log"
Jul 29 20:27:13.027212 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:13.027189    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c757d79-g5bhl_b892161e-6e80-46d2-bb45-64e5aefb98dc/prom-label-proxy/0.log"
Jul 29 20:27:13.049062 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:13.049039    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c757d79-g5bhl_b892161e-6e80-46d2-bb45-64e5aefb98dc/kube-rbac-proxy-rules/0.log"
Jul 29 20:27:13.072861 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:13.072794    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c757d79-g5bhl_b892161e-6e80-46d2-bb45-64e5aefb98dc/kube-rbac-proxy-metrics/0.log"
Jul 29 20:27:14.199950 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.199913    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"]
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1209f71b-a54a-4c51-b968-017523b38c98" containerName="payload-pre-processing"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200286    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1209f71b-a54a-4c51-b968-017523b38c98" containerName="payload-pre-processing"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200296    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" containerName="payload-pre-processing"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200301    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" containerName="payload-pre-processing"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200314    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1bc531-5c5d-43f7-a061-60ff3c59d148" containerName="payload-processing"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200320    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1bc531-5c5d-43f7-a061-60ff3c59d148" containerName="payload-processing"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" containerName="maas-api"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200331    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" containerName="maas-api"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200350    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d182f2-dbfa-4186-9377-69ba6d0c27aa" containerName="maas-api"
Jul 29 20:27:14.200357 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200358    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d182f2-dbfa-4186-9377-69ba6d0c27aa" containerName="maas-api"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200378    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c49992e-de73-4a27-9bac-59f679ac5b9c" containerName="payload-pre-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200386    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c49992e-de73-4a27-9bac-59f679ac5b9c" containerName="payload-pre-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200406    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a059d6-879c-411e-a317-63d7185089a9" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200414    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a059d6-879c-411e-a317-63d7185089a9" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200423    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53336460-7a6b-4737-84f6-06248ed5eb4c" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200431    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53336460-7a6b-4737-84f6-06248ed5eb4c" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200443    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" containerName="main"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200450    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" containerName="main"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200463    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8013b929-27b9-4574-a68b-2d2c4f59a10e" containerName="maas-api"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200470    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8013b929-27b9-4574-a68b-2d2c4f59a10e" containerName="maas-api"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200554    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c49992e-de73-4a27-9bac-59f679ac5b9c" containerName="payload-pre-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200567    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1209f71b-a54a-4c51-b968-017523b38c98" containerName="payload-pre-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200588    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a059d6-879c-411e-a317-63d7185089a9" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200593    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8013b929-27b9-4574-a68b-2d2c4f59a10e" containerName="maas-api"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200600    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53336460-7a6b-4737-84f6-06248ed5eb4c" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200609    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1bc531-5c5d-43f7-a061-60ff3c59d148" containerName="payload-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200617    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c505fe5-b9fa-4a54-9b6a-2a5cd3d9419b" containerName="payload-pre-processing"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200624    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c787f1d8-3e11-4b94-8a0a-ab3b3ce6b493" containerName="maas-api"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200630    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d182f2-dbfa-4186-9377-69ba6d0c27aa" containerName="maas-api"
Jul 29 20:27:14.200792 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.200635    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ee90f5-9d95-4a99-afa2-06bd55bb5ff8" containerName="main"
Jul 29 20:27:14.203712 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.203693    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.205988 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.205968    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cq5dz\"/\"openshift-service-ca.crt\""
Jul 29 20:27:14.206109 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.205968    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cq5dz\"/\"default-dockercfg-8c6m4\""
Jul 29 20:27:14.206354 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.206339    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cq5dz\"/\"kube-root-ca.crt\""
Jul 29 20:27:14.213079 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.213054    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"]
Jul 29 20:27:14.258967 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.258937    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-podres\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.259148 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.258978    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-sys\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.259148 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.259007    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhgr\" (UniqueName: \"kubernetes.io/projected/d48154ab-1c41-4225-976a-3a5fccefea0a-kube-api-access-4lhgr\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.259278 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.259168    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-proc\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.259278 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.259220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-lib-modules\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360008 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.359973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-proc\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360008 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-lib-modules\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-podres\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-sys\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhgr\" (UniqueName: \"kubernetes.io/projected/d48154ab-1c41-4225-976a-3a5fccefea0a-kube-api-access-4lhgr\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360117    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-proc\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360167    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-lib-modules\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360187    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-sys\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.360239 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.360209    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d48154ab-1c41-4225-976a-3a5fccefea0a-podres\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.368798 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.368772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhgr\" (UniqueName: \"kubernetes.io/projected/d48154ab-1c41-4225-976a-3a5fccefea0a-kube-api-access-4lhgr\") pod \"perf-node-gather-daemonset-5xzk8\" (UID: \"d48154ab-1c41-4225-976a-3a5fccefea0a\") " pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.514507 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.514428    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:14.647049 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:14.647018    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"]
Jul 29 20:27:14.650363 ip-10-0-138-139 kubenswrapper[2555]: W0729 20:27:14.650331    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd48154ab_1c41_4225_976a_3a5fccefea0a.slice/crio-38137843ac3d6e2395156da5309c1983f860f5f41d751b2cfe6d09cd43d16013 WatchSource:0}: Error finding container 38137843ac3d6e2395156da5309c1983f860f5f41d751b2cfe6d09cd43d16013: Status 404 returned error can't find the container with id 38137843ac3d6e2395156da5309c1983f860f5f41d751b2cfe6d09cd43d16013
Jul 29 20:27:15.106154 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:15.106111    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8" event={"ID":"d48154ab-1c41-4225-976a-3a5fccefea0a","Type":"ContainerStarted","Data":"44304b617b917f64007efd6ef09c0b93a8fc29d99390c9847e6fd02410a4bcb9"}
Jul 29 20:27:15.106361 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:15.106299    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:15.106361 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:15.106326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8" event={"ID":"d48154ab-1c41-4225-976a-3a5fccefea0a","Type":"ContainerStarted","Data":"38137843ac3d6e2395156da5309c1983f860f5f41d751b2cfe6d09cd43d16013"}
Jul 29 20:27:15.123130 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:15.123066    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8" podStartSLOduration=1.123049063 podStartE2EDuration="1.123049063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:27:15.122463792 +0000 UTC m=+3441.080958685" watchObservedRunningTime="2026-07-29 20:27:15.123049063 +0000 UTC m=+3441.081543958"
Jul 29 20:27:15.525267 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:15.525234    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-xnzj9_26f89cc3-3bde-4629-b636-f83c62f29c3f/download-server/0.log"
Jul 29 20:27:17.270759 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:17.270728    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cddmj_8838db34-5e79-4ad7-954d-956d6d35ed2f/dns/0.log"
Jul 29 20:27:17.297337 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:17.297308    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cddmj_8838db34-5e79-4ad7-954d-956d6d35ed2f/kube-rbac-proxy/0.log"
Jul 29 20:27:17.371971 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:17.371934    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6g6hg_16d4adba-8f03-4c88-96d3-cfc9449182f7/dns-node-resolver/0.log"
Jul 29 20:27:18.029084 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:18.029058    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-577699577-shb6z_9aec82e6-db34-4e43-8bb4-8189e02c2a06/registry/0.log"
Jul 29 20:27:18.057691 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:18.057662    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hw7vt_0c822540-948e-4f1e-8189-00f4c8bff2b8/node-ca/0.log"
Jul 29 20:27:19.265419 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:19.265383    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cc65fd5df-njhfl_c9e686ba-1b46-4989-b6e5-ba3adec565c4/kube-auth-proxy/0.log"
Jul 29 20:27:19.428807 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:19.428778    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hwgbd_9d222932-eac3-4e73-954b-af06b118b1d5/payload-processing/0.log"
Jul 29 20:27:20.358365 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:20.358330    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wzmhh_cd75f887-a524-4013-a82a-6d959ed55c52/serve-healthcheck-canary/0.log"
Jul 29 20:27:20.877272 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:20.877240    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlkwn_639a026f-f76d-4c50-ad25-20f56fe83839/kube-rbac-proxy/0.log"
Jul 29 20:27:20.903723 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:20.903691    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlkwn_639a026f-f76d-4c50-ad25-20f56fe83839/exporter/0.log"
Jul 29 20:27:20.927206 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:20.927179    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlkwn_639a026f-f76d-4c50-ad25-20f56fe83839/extractor/0.log"
Jul 29 20:27:21.120105 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:21.120078    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cq5dz/perf-node-gather-daemonset-5xzk8"
Jul 29 20:27:23.955974 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:23.955935    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c74855f4-dmcx7_6fa5223b-f9d0-41d4-92af-ffa4896de0f9/manager/0.log"
Jul 29 20:27:24.106478 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:24.106450    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d75b7b96-lbglf_684e45cf-b6db-448c-b7ce-73fd08a66308/manager/0.log"
Jul 29 20:27:24.134248 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:24.134212    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d75b7b96-lbglf_684e45cf-b6db-448c-b7ce-73fd08a66308/copy-manifests/0.log"
Jul 29 20:27:25.044445 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:25.044399    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cc65fd5df-njhfl_c9e686ba-1b46-4989-b6e5-ba3adec565c4/kube-auth-proxy/0.log"
Jul 29 20:27:25.171163 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:25.171130    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hwgbd_9d222932-eac3-4e73-954b-af06b118b1d5/payload-processing/0.log"
Jul 29 20:27:25.645473 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:25.645443    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-fcbbc6665-ngk5f_45280a7d-00b8-42d6-a6d9-66d239c2ea22/manager/0.log"
Jul 29 20:27:26.808142 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:26.808112    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cc65fd5df-njhfl_c9e686ba-1b46-4989-b6e5-ba3adec565c4/kube-auth-proxy/0.log"
Jul 29 20:27:26.934415 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:26.934378    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hwgbd_9d222932-eac3-4e73-954b-af06b118b1d5/payload-processing/0.log"
Jul 29 20:27:30.749661 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:30.749628    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-spp7q_b8392df1-96e0-490d-b551-54fef247c2fd/migrator/0.log"
Jul 29 20:27:30.771374 ip-10-0-138-139 kubenswrapper[2555]: I0729 20:27:30.771344    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-spp7q_b8392df1-96e0-490d-b551-54fef247c2fd/graceful-termination/0.log"