Jul 27 23:16:14.949680 ip-10-0-136-102 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 23:16:15.272908 ip-10-0-136-102 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 23:16:15.272908 ip-10-0-136-102 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 23:16:15.272908 ip-10-0-136-102 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 23:16:15.272908 ip-10-0-136-102 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 23:16:15.272908 ip-10-0-136-102 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 23:16:15.273756 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.273667    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 23:16:15.279434 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279415    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:15.279434 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279430    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:15.279434 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279435    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:15.279434 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279439    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:15.279434 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279443    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279447    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279450    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279458    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279464    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279480    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279485    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279490    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279494    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279497    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279501    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279505    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279509    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279512    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279516    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279520    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279524    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279528    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279531    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:15.279747 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279536    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279539    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279543    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279548    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279552    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279555    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279560    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279564    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279568    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279572    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279576    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279580    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279585    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279591    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279595    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279600    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279605    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279609    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279613    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279619    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279624    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:15.280545 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279628    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279631    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279636    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279640    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279644    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279648    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279652    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279656    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279661    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279665    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279669    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279673    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279677    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279681    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279686    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279690    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279694    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279698    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279703    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:15.281467 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279707    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279715    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279721    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279726    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279731    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279736    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279741    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279746    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279751    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279755    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279760    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279765    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279771    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279775    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279779    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279784    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279792    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279798    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279802    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:15.282245 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279807    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279811    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279815    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279819    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279823    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279827    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279832    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279836    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279840    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279844    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279849    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279852    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279857    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279861    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279865    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279869    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.279873    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281092    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281104    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:15.283075 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281109    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281115    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281120    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281125    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281131    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281135    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281140    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281145    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281150    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281153    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281158    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281162    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281167    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281171    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281176    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281180    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281184    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281188    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281192    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281196    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:15.283933 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281200    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281204    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281208    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281213    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281217    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281221    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281225    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281229    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281233    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281237    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281241    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281247    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281251    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281256    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281260    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281264    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281268    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281272    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281276    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:15.284478 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281284    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281290    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281295    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281299    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281304    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281308    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281314    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281319    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281323    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281327    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281331    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281336    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281340    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281345    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281367    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281372    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281377    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281380    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281385    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281392    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:15.284973 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281398    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281403    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281407    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281411    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281415    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281420    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281424    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281428    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281432    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281436    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281441    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281445    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281449    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281453    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281458    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281462    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281466    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281470    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281474    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:15.285553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281478    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281482    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281486    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281490    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281494    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281499    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281503    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281507    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281512    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281516    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281520    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281524    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281528    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281532    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281537    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281541    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281545    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281550    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.281554    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:15.286020 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282151    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282159    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282163    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282168    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282172    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282176    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282180    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282184    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282189    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282194    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282198    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282202    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282207    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282211    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282215    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282220    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282224    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282229    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282233    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282245    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:15.286501 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282249    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282253    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282257    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282261    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282268    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282274    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282279    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282284    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282288    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282292    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282297    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282302    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282306    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282310    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282314    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282319    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282323    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282327    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282331    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:15.286986 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282335    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282339    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282344    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282366    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282371    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282375    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282380    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282384    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282388    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282392    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282396    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282401    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282405    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282409    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282414    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282417    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282421    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282425    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282429    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:15.287462 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282433    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282437    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282441    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282445    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282450    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282454    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282458    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282462    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282466    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282470    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282475    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282480    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282484    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282489    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282493    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282499    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282506    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282511    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282515    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282519    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:15.287923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282524    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282529    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282534    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282538    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282542    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282546    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282550    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282556    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282560    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282567    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282572    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282576    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282581    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282586    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282591    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282595    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282602    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282606    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282611    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:15.288423 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282616    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.282620    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282727    2570 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282737    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282746    2570 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282753    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282760    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282766    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282773    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282779    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282786    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282791    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282796    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282802    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282807    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282812    2570 flags.go:64] FLAG: --cgroup-root=""
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282817    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282822    2570 flags.go:64] FLAG: --client-ca-file=""
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282826    2570 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282830    2570 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282837    2570 flags.go:64] FLAG: --cluster-domain=""
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282842    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282847    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282852    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 23:16:15.288881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282857    2570 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282863    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282867    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282873    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282878    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282887    2570 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282891    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282896    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282901    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282906    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282912    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282917    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282921    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282926    2570 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282930    2570 flags.go:64] FLAG: --enable-server="true"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282935    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282941    2570 flags.go:64] FLAG: --event-burst="100"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282945    2570 flags.go:64] FLAG: --event-qps="50"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282950    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282955    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282961    2570 flags.go:64] FLAG: --eviction-hard=""
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282967    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282971    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282976    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282981    2570 flags.go:64] FLAG: --eviction-soft=""
Jul 27 23:16:15.289478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282986    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282990    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.282995    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283000    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283005    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283009    2570 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283014    2570 flags.go:64] FLAG: --feature-gates=""
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283020    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283024    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283029    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283034    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283039    2570 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283045    2570 flags.go:64] FLAG: --help="false"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283050    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283055    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283060    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283064    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283070    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283075    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283079    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283083    2570 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283087    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283092    2570 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 23:16:15.290105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283096    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283101    2570 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283106    2570 flags.go:64] FLAG: --kube-reserved=""
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283111    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283116    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283122    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283127    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283132    2570 flags.go:64] FLAG: --lock-file=""
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283136    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283141    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283145    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283154    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283159    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283163    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283188    2570 flags.go:64] FLAG: --logging-format="text"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283195    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283201    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283206    2570 flags.go:64] FLAG: --manifest-url=""
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283211    2570 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283217    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283222    2570 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283233    2570 flags.go:64] FLAG: --max-pods="110"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283237    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283242    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283247    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 23:16:15.290772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283252    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283256    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283261    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283265    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283276    2570 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283281    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283287    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283292    2570 flags.go:64] FLAG: --pod-cidr=""
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283297    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283305    2570 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283310    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283315    2570 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283320    2570 flags.go:64] FLAG: --port="10250"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283325    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283330    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0380cb38dea68a5dd"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283335    2570 flags.go:64] FLAG: --qos-reserved=""
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283340    2570 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283344    2570 flags.go:64] FLAG: --register-node="true"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283369    2570 flags.go:64] FLAG: --register-with-taints=""
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283375    2570 flags.go:64] FLAG: --registry-burst="10"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283380    2570 flags.go:64] FLAG: --registry-qps="5"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283384    2570 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283389    2570 flags.go:64] FLAG: --reserved-memory=""
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283395    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283399    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 23:16:15.291645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283404    2570 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283408    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283412    2570 flags.go:64] FLAG: --runonce="false"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283417    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283424    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283429    2570 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283434    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283438    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283443    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283448    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283452    2570 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283457    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283461    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283467    2570 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283473    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283478    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283483    2570 flags.go:64] FLAG: --system-cgroups=""
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283487    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283496    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283501    2570 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283506    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283512    2570 flags.go:64] FLAG: --tls-min-version=""
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283517    2570 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283521    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283526    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 23:16:15.292455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283531    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283536    2570 flags.go:64] FLAG: --v="2"
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283542    2570 flags.go:64] FLAG: --version="false"
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283549    2570 flags.go:64] FLAG: --vmodule=""
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283554    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.283560    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283744    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283751    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283756    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283761    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283765    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283772    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283776    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283780    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283785    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283789    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283794    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283798    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283802    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283806    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:15.293069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283812    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283816    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283820    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283824    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283830    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283834    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283838    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283842    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283847    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283851    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283855    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283860    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283864    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283868    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283872    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283876    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283880    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283884    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283888    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283892    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:15.293561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283896    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283901    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283905    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283910    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283914    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283921    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283928    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283934    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283938    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283943    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283947    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283952    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283957    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.283981    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284013    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284018    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284022    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284025    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284029    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:15.294047 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284032    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284035    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284037    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284040    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284042    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284045    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284048    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284051    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284054    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284057    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284059    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284062    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284065    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284067    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284070    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284072    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284075    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284078    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284080    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284083    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:15.294588 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284085    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284088    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284091    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284093    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284096    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284098    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284101    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284104    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284113    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284116    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284118    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284121    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284123    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284125    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284128    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284130    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284133    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284135    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284137    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284141    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:15.295063 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284143    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284145    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284148    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284150    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284153    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.284155    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.284716    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293001    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293020    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293039    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293045    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293107    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293111    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293115    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:15.295565 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293119    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293123    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293126    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293129    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293132    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293135    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293138    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293142    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293146    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293149    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293153    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293156    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293158    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293161    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293163    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293166    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293168    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293171    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293173    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:15.295908 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293176    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293178    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293181    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293184    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293186    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293189    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293191    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293194    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293196    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293199    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293202    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293204    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293207    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293209    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293211    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293213    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293216    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293218    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293221    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293224    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293226    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:15.296391 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293229    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293233    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293236    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293239    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293242    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293245    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293247    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293250    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293252    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293255    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293257    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293260    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293263    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293265    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293267    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293270    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293272    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293274    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293277    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:15.296889 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293279    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293282    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293284    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293286    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293289    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293291    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293293    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293296    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293298    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293301    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293304    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293307    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293309    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293312    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293315    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293317    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293319    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293323    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293325    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:15.297421 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293328    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293330    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293332    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293335    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293337    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293340    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293342    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293344    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293346    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293349    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293365    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293368    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293371    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293373    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293376    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293378    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293381    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:15.297923 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293383    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293388    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293504    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293508    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293511    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293514    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293517    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293520    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293522    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293525    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293527    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293530    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293532    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293535    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293538    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 23:16:15.298337 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293541    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293543    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293545    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293548    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293550    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293552    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293555    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293558    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293560    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293563    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293565    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293568    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293570    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293573    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293576    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293580    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293583    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293586    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293589    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293591    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 23:16:15.298723 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293594    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293596    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293599    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293602    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293604    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293606    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293609    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293611    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293614    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293616    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293619    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293621    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293624    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293627    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293629    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293632    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293635    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293639    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293641    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 23:16:15.299190 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293644    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293647    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293649    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293652    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293654    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293656    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293659    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293661    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293663    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293666    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293668    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293670    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293673    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293675    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293678    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293680    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293683    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293686    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293688    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293690    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 23:16:15.299657 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293693    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293696    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293698    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293700    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293703    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293705    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293708    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293711    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293715    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293718    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293721    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293723    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293726    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293730    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293732    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293735    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293738    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293740    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293742    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293745    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 23:16:15.300130 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293747    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293750    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293752    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293754    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293757    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293759    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:15.293762    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293769    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.293878    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.296578    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.297697    2570 server.go:1019] "Starting client certificate rotation"
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.297798    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.297837    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 23:16:15.300628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.299810    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 23:16:15.301027 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.301013    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 23:16:15.310055 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.310036    2570 log.go:25] "Validated CRI v1 runtime API"
Jul 27 23:16:15.313628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.313612    2570 log.go:25] "Validated CRI v1 image API"
Jul 27 23:16:15.315000 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.314986    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 23:16:15.318795 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.318772    2570 fs.go:135] Filesystem UUIDs: map[0b35f824-429a-4352-8ff5-5a6e775b59f1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b9349167-2500-48b2-919f-0a563d47c044:/dev/nvme0n1p4]
Jul 27 23:16:15.318869 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.318795    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 23:16:15.323398 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.323376    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 23:16:15.326011 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.324370    2570 manager.go:216] Machine: {Timestamp:2026-07-27 23:16:15.32321826 +0000 UTC m=+0.274561648 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108832 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea67d4500b28c98d1c5a77d174f1e SystemUUID:ec2ea67d-4500-b28c-98d1-c5a77d174f1e BootID:0a814a37-c503-4a41-8de3-f460374ac3aa Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a9:f2:22:cd:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a9:f2:22:cd:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:07:81:de:1d:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 23:16:15.326011 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.325243    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 23:16:15.326011 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.325395    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 23:16:15.326667 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.326628    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 23:16:15.327067 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.326678    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 23:16:15.327593 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.327580    2570 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 23:16:15.327632 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.327596    2570 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 23:16:15.327632 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.327605    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 23:16:15.328204 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.328194    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 23:16:15.328997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.328988    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 23:16:15.331655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331644    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 23:16:15.331770 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331760    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 23:16:15.331806 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331789    2570 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 23:16:15.331806 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331799    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 23:16:15.331884 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331814    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 23:16:15.331884 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331822    2570 kubelet.go:398] "Adding apiserver pod source"
Jul 27 23:16:15.331884 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.331830    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 23:16:15.332210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.332199    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 23:16:15.332243 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.332218    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 23:16:15.334659 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.334641    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 23:16:15.336570 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.336554    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 23:16:15.336636 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.336586    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 23:16:15.338022 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338009    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338025    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338031    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338036    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338045    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338051    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338059    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338064    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338071    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338077    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338085    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 23:16:15.338101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338091    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 23:16:15.338674 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338660    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 23:16:15.338674 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.338676    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 23:16:15.342245 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.342223    2570 server.go:1268] "Started kubelet"
Jul 27 23:16:15.342367 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.342316    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 23:16:15.342518 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.342501    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 23:16:15.342657 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.342600    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 23:16:15.342762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.342707    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 23:16:15.343194 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.343163    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 23:16:15.343194 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.343162    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 23:16:15.343183 ip-10-0-136-102 systemd[1]: Started Kubernetes Kubelet.
Jul 27 23:16:15.344263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.344246    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 23:16:15.344601 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.344586    2570 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 23:16:15.345052 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.345031    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-88npn"
Jul 27 23:16:15.347127 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.346395    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-102.ec2.internal.18c648a4e45e2bf7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-102.ec2.internal,UID:ip-10-0-136-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-102.ec2.internal,},FirstTimestamp:2026-07-27 23:16:15.342193655 +0000 UTC m=+0.293537032,LastTimestamp:2026-07-27 23:16:15.342193655 +0000 UTC m=+0.293537032,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-102.ec2.internal,}"
Jul 27 23:16:15.347524 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.347504    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 23:16:15.350729 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.350711    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 23:16:15.350790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.350721    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 23:16:15.350856 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.350838    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.351376 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.351340    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 23:16:15.351503 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.351468    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 23:16:15.351576 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.351506    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 23:16:15.351820 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.351641    2570 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 23:16:15.351820 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.351656    2570 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 23:16:15.352779 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.352747    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-88npn"
Jul 27 23:16:15.352976 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.352959    2570 factory.go:55] Registering systemd factory
Jul 27 23:16:15.353091 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.353009    2570 factory.go:223] Registration of the systemd container factory successfully
Jul 27 23:16:15.353343 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.353311    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 23:16:15.353533 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.353462    2570 factory.go:153] Registering CRI-O factory
Jul 27 23:16:15.353533 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.353536    2570 factory.go:223] Registration of the crio container factory successfully
Jul 27 23:16:15.353682 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.353475    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 23:16:15.353682 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.353622    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 23:16:15.353682 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.353648    2570 factory.go:103] Registering Raw factory
Jul 27 23:16:15.353682 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.353664    2570 manager.go:1196] Started watching for new ooms in manager
Jul 27 23:16:15.354963 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.354948    2570 manager.go:349] Starting recovery of all containers
Jul 27 23:16:15.362683 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.362635    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 23:16:15.366450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.366431    2570 manager.go:354] Recovery completed
Jul 27 23:16:15.371181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.371168    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.373094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373069    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.373153 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373103    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.373153 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373118    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.373659 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373644    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 23:16:15.373659 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373656    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 23:16:15.373767 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373658    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.373767 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.373674    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 23:16:15.374364 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.374335    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.374421 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.374383    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.374421 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.374393    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.376165 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.376154    2570 policy_none.go:49] "None policy: Start"
Jul 27 23:16:15.376213 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.376169    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 23:16:15.376213 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.376179    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 23:16:15.378167 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.378156    2570 policy_none.go:47] "Start"
Jul 27 23:16:15.412931 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.412914    2570 manager.go:337] "Starting Device Plugin manager"
Jul 27 23:16:15.413013 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.412950    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 23:16:15.413013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.412961    2570 server.go:87] "Starting device plugin registration server"
Jul 27 23:16:15.413247 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.413232    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 23:16:15.413329 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.413249    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 23:16:15.413393 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.413333    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 23:16:15.413454 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.413429    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 23:16:15.413454 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.413439    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 23:16:15.414018 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.413990    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 23:16:15.414102 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.414039    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.513489 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.513460    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.514511    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.514556    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.514573    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.514613    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.515501    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.515549    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.515577    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 23:16:15.515736 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.515616    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 23:16:15.517483 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.517466    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:15.522109 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.522093    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.522189 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.522116    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-102.ec2.internal\": node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.566555 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.566484    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.615753 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.615705    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"]
Jul 27 23:16:15.615851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.615805    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.617393 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.617377    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.617450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.617410    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.617450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.617419    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.619797 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.619785    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.619937 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.619922    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.619980 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.619964    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.620495 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.620478    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.620495 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.620492    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.620645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.620510    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.620645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.620514    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.620645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.620520    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.620645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.620529    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.622763 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.622747    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.622857 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.622774    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 23:16:15.623419 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.623404    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 23:16:15.623490 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.623431    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 23:16:15.623490 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.623444    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID"
Jul 27 23:16:15.640871 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.640851    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-102.ec2.internal\" not found" node="ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.644653 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.644639    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-102.ec2.internal\" not found" node="ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.653639 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.653624    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.653687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.653648    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.653687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.653664    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d1c0db1c0eb2e7cc88fd5e78b49f45-config\") pod \"kube-apiserver-proxy-ip-10-0-136-102.ec2.internal\" (UID: \"76d1c0db1c0eb2e7cc88fd5e78b49f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.667283 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.667267    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.754127 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.754095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.754253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.754132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.754253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.754161    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d1c0db1c0eb2e7cc88fd5e78b49f45-config\") pod \"kube-apiserver-proxy-ip-10-0-136-102.ec2.internal\" (UID: \"76d1c0db1c0eb2e7cc88fd5e78b49f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.754253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.754189    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.754253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.754205    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.754253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.754198    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d1c0db1c0eb2e7cc88fd5e78b49f45-config\") pod \"kube-apiserver-proxy-ip-10-0-136-102.ec2.internal\" (UID: \"76d1c0db1c0eb2e7cc88fd5e78b49f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.768213 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.768187    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.868963 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.868890    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:15.944133 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.944095    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.947507 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:15.947491    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:15.970008 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:15.969976    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.070574 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:16.070534    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.171019 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:16.170990    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.272106 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:16.272073    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.297477 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.297438    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 23:16:16.297949 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.297631    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 23:16:16.365959 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.365915    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 23:11:15 +0000 UTC" deadline="2028-05-11 01:10:53.791517492 +0000 UTC"
Jul 27 23:16:16.365959 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.365947    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15673h54m37.425573877s"
Jul 27 23:16:16.372890 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:16.372869    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.473020 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:16.472995    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.573204 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:16.573168    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found"
Jul 27 23:16:16.581445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.581404    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:16.597090 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:16.597054    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d1c0db1c0eb2e7cc88fd5e78b49f45.slice/crio-bf6085a71126c39397e99b9350be3c3019e898921de95842c086ee0ff5b4cd0b WatchSource:0}: Error finding container bf6085a71126c39397e99b9350be3c3019e898921de95842c086ee0ff5b4cd0b: Status 404 returned error can't find the container with id bf6085a71126c39397e99b9350be3c3019e898921de95842c086ee0ff5b4cd0b
Jul 27 23:16:16.602997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.602974    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:16:16.609615 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:16.609583    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8787e14cefbe48afd866da4ad18abf46.slice/crio-034ef5017555e6e627ef2b4f941a34c601770ae9dd2f9fa8958b2df6b2188877 WatchSource:0}: Error finding container 034ef5017555e6e627ef2b4f941a34c601770ae9dd2f9fa8958b2df6b2188877: Status 404 returned error can't find the container with id 034ef5017555e6e627ef2b4f941a34c601770ae9dd2f9fa8958b2df6b2188877
Jul 27 23:16:16.620437 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.620403    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:16.652676 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.652631    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:16.664253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.664221    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 23:16:16.665302 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.665284    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"
Jul 27 23:16:16.674186 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.674160    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 23:16:16.958436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:16.958399    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:17.177238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.177202    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 23:16:17.334594 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.334498    2570 apiserver.go:52] "Watching apiserver"
Jul 27 23:16:17.342383 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.342340    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 23:16:17.342853 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.342823    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2drhj","openshift-dns/node-resolver-pc986","openshift-multus/multus-additional-cni-plugins-hdswt","openshift-network-operator/iptables-alerter-gwfj6","kube-system/konnectivity-agent-7ktzp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb","openshift-image-registry/node-ca-nkb9d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal","openshift-multus/multus-69bq9","openshift-multus/network-metrics-daemon-vbgm4","openshift-network-diagnostics/network-check-target-cd7ss","openshift-ovn-kubernetes/ovnkube-node-gpjmg","kube-system/global-pull-secret-syncer-hvt44"]
Jul 27 23:16:17.346113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.346093    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.349209 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.349143    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 23:16:17.349209 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.349165    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x8hlr\""
Jul 27 23:16:17.349209 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.349179    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.349458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.349181    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.351011 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.350556    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.351365 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.351342    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 23:16:17.353538 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.353514    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.355205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.354815    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.355205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.355017    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 23:16:17.355395 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.355202    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5n7w\""
Jul 27 23:16:17.355395 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.355373    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 23:16:17.355536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.355451    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.355594 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.355569    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 23:16:17.355594 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.355217    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 23:16:17.356807 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.356412    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.356807 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.356526    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fxkk\""
Jul 27 23:16:17.356807 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.356632    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.357005 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.356882    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.358734 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.358385    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.359757 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.359737    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 23:16:17.360271 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.360251    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zwrvv\""
Jul 27 23:16:17.360603 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.360582    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 23:16:17.360830 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.360815    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.361495    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.361642    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.361695    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.361783    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.361933    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s6qlb\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.361941    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.362110 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.362078    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.363356    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.363457    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.363994    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-cni-bin\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a32c919-8faa-44fd-9e91-d696e209a766-hosts-file\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364206    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-slash\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-socket-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364348    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-registration-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-device-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.364458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364420    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-var-lib-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beecc250-ce58-4af8-a233-f23355d0d0f5-ovn-node-metrics-cert\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364501    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-ovnkube-config\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-env-overrides\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a32c919-8faa-44fd-9e91-d696e209a766-tmp-dir\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364679    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364834    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-etc-selinux\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-sys-fs\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.364989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364897    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-kubelet\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.364940    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-etc-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-log-socket\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-cni-netd\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhql9\" (UniqueName: \"kubernetes.io/projected/d05eeaa5-6b29-4da3-8c1d-d487592f8265-kube-api-access-bhql9\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365152    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-ovnkube-script-lib\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365185    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sqz\" (UniqueName: \"kubernetes.io/projected/4a32c919-8faa-44fd-9e91-d696e209a766-kube-api-access-s7sqz\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-systemd-units\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365300    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365327    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-ovn\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365372 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-run-netns\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365875 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365392    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-systemd\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365875 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365424    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365875 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgt2m\" (UniqueName: \"kubernetes.io/projected/beecc250-ce58-4af8-a233-f23355d0d0f5-kube-api-access-xgt2m\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.365875 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.365505    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-node-log\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.368295 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.366135    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 23:16:17.368295 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.366830    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 23:16:17.368295 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.366842    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-st9mz\""
Jul 27 23:16:17.368520 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.368500    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 23:16:17.370976 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.370579    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.373785 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.373270    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.375606 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.375576    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.375725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.375704    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4xtv\""
Jul 27 23:16:17.375944 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.375922    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:16:17.376015 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.375955    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.376254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.376236    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v7j7m\""
Jul 27 23:16:17.376328 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.376236    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 23:16:17.379506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.379485    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.379611 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.379597    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 23:16:17.382417 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.382393    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 23:16:17.382521 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.382496    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98zpz\""
Jul 27 23:16:17.383145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.383120    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9qgw"
Jul 27 23:16:17.384161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.384117    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:17.384268 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.384212    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:17.387013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.386985    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:17.387107 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.387075    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:17.393420 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.393399    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9qgw"
Jul 27 23:16:17.452815 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.452793    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 23:16:17.466600 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466568    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v752r\" (UniqueName: \"kubernetes.io/projected/3e0e3824-2d46-4521-a962-53e7ffce99ca-kube-api-access-v752r\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.466753 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466614    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-daemon-config\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.466753 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466649    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-cni-bin\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.466753 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466679    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8763dd4f-6c77-4916-88b7-01f454d22603-kubelet-config\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.466753 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466708    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmq6s\" (UniqueName: \"kubernetes.io/projected/b48ab19b-f661-4ae8-84fe-96685d29fcfb-kube-api-access-lmq6s\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.466753 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466735    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-modprobe-d\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.466927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466759    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e0e3824-2d46-4521-a962-53e7ffce99ca-tmp\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.466927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466786    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-conf-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.466927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.466927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466848    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-socket-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.466927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466888    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-registration-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466917    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-var-lib-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466962    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.466992    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysctl-conf\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467019    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467048    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-ovnkube-config\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467086    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.467141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467114    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-tuned\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467142    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-system-cni-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467171    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467203    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-etc-selinux\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-kubelet\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-etc-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467287    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-log-socket\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467316    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-cni-netd\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467351    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-iptables-alerter-script\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.467451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467410    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhql9\" (UniqueName: \"kubernetes.io/projected/d05eeaa5-6b29-4da3-8c1d-d487592f8265-kube-api-access-bhql9\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysconfig\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-cnibin\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-etc-kubernetes\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467542    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rrb\" (UniqueName: \"kubernetes.io/projected/b3acd34a-68bc-472f-a6e1-8169dbed08db-kube-api-access-48rrb\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-systemd-units\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467611    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-ovn\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467673    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-system-cni-dir\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpvl\" (UniqueName: \"kubernetes.io/projected/72751d1b-30eb-40e1-b971-e83409fedf08-kube-api-access-5lpvl\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467749    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-run-netns\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467778    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-systemd\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467808    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-cnibin\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467835    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b48ab19b-f661-4ae8-84fe-96685d29fcfb-serviceca\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467862    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-lib-modules\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467892    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-k8s-cni-cncf-io\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-netns\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468039 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.467977    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-node-log\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-hostroot\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468032    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-host-slash\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468061    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a32c919-8faa-44fd-9e91-d696e209a766-hosts-file\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-cni-binary-copy\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3debe752-2e41-4cd7-9265-80843637eb17-konnectivity-ca\") pod \"konnectivity-agent-7ktzp\" (UID: \"3debe752-2e41-4cd7-9265-80843637eb17\") " pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468148    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b48ab19b-f661-4ae8-84fe-96685d29fcfb-host\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-host\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-cni-binary-copy\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-slash\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468278    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-device-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beecc250-ce58-4af8-a233-f23355d0d0f5-ovn-node-metrics-cert\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-env-overrides\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468361    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a32c919-8faa-44fd-9e91-d696e209a766-tmp-dir\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3debe752-2e41-4cd7-9265-80843637eb17-agent-certs\") pod \"konnectivity-agent-7ktzp\" (UID: \"3debe752-2e41-4cd7-9265-80843637eb17\") " pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468413    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysctl-d\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468445    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-os-release\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.468993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-sys-fs\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sqz\" (UniqueName: \"kubernetes.io/projected/4a32c919-8faa-44fd-9e91-d696e209a766-kube-api-access-s7sqz\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468530    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-os-release\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8763dd4f-6c77-4916-88b7-01f454d22603-dbus\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468586    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-run\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468647    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-systemd\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-var-lib-kubelet\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468714    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-ovnkube-script-lib\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-kubernetes\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468774    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-socket-dir-parent\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468802    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-cni-bin\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468834    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468868    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-sys\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-cni-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-cni-multus\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-kubelet\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.468993    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlcl\" (UniqueName: \"kubernetes.io/projected/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-kube-api-access-9hlcl\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.469860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469023    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469054    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgt2m\" (UniqueName: \"kubernetes.io/projected/beecc250-ce58-4af8-a233-f23355d0d0f5-kube-api-access-xgt2m\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-multus-certs\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469144    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnb2\" (UniqueName: \"kubernetes.io/projected/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-kube-api-access-dmnb2\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469260    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-cni-bin\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469382    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-registration-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469566    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-var-lib-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a32c919-8faa-44fd-9e91-d696e209a766-hosts-file\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469777    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-slash\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.469835    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-device-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470208    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470265    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-ovnkube-config\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470529    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-etc-selinux\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-kubelet\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.470725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470637    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-etc-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470684    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-log-socket\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.470685    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-ovnkube-script-lib\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471054    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beecc250-ce58-4af8-a233-f23355d0d0f5-env-overrides\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471064    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-systemd-units\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471127    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-ovn\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471196    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-run-netns\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471246    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-systemd\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471296    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a32c919-8faa-44fd-9e91-d696e209a766-tmp-dir\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471344    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-run-openvswitch\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471343    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-node-log\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471450    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-sys-fs\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.471605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471458    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-cni-netd\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.472262 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471911    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d05eeaa5-6b29-4da3-8c1d-d487592f8265-socket-dir\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.472262 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.471906    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beecc250-ce58-4af8-a233-f23355d0d0f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.475240 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.475214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beecc250-ce58-4af8-a233-f23355d0d0f5-ovn-node-metrics-cert\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.480198 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.480149    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgt2m\" (UniqueName: \"kubernetes.io/projected/beecc250-ce58-4af8-a233-f23355d0d0f5-kube-api-access-xgt2m\") pod \"ovnkube-node-gpjmg\" (UID: \"beecc250-ce58-4af8-a233-f23355d0d0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.481277 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.480823    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sqz\" (UniqueName: \"kubernetes.io/projected/4a32c919-8faa-44fd-9e91-d696e209a766-kube-api-access-s7sqz\") pod \"node-resolver-pc986\" (UID: \"4a32c919-8faa-44fd-9e91-d696e209a766\") " pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.483575 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.483533    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhql9\" (UniqueName: \"kubernetes.io/projected/d05eeaa5-6b29-4da3-8c1d-d487592f8265-kube-api-access-bhql9\") pod \"aws-ebs-csi-driver-node-6tnpb\" (UID: \"d05eeaa5-6b29-4da3-8c1d-d487592f8265\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.522874 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.522806    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" event={"ID":"8787e14cefbe48afd866da4ad18abf46","Type":"ContainerStarted","Data":"034ef5017555e6e627ef2b4f941a34c601770ae9dd2f9fa8958b2df6b2188877"}
Jul 27 23:16:17.524184 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.524158    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" event={"ID":"76d1c0db1c0eb2e7cc88fd5e78b49f45","Type":"ContainerStarted","Data":"bf6085a71126c39397e99b9350be3c3019e898921de95842c086ee0ff5b4cd0b"}
Jul 27 23:16:17.570325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570287    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-daemon-config\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.570535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570338    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8763dd4f-6c77-4916-88b7-01f454d22603-kubelet-config\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.570535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570366    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmq6s\" (UniqueName: \"kubernetes.io/projected/b48ab19b-f661-4ae8-84fe-96685d29fcfb-kube-api-access-lmq6s\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.570535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570413    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-modprobe-d\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.570535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570440    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e0e3824-2d46-4521-a962-53e7ffce99ca-tmp\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.570535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-conf-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.570535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570495    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570554    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysctl-conf\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570588    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-tuned\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570618    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-system-cni-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570655    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-iptables-alerter-script\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570672    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysconfig\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-cnibin\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570703    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-etc-kubernetes\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570732    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rrb\" (UniqueName: \"kubernetes.io/projected/b3acd34a-68bc-472f-a6e1-8169dbed08db-kube-api-access-48rrb\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:17.570823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-system-cni-dir\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570827    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpvl\" (UniqueName: \"kubernetes.io/projected/72751d1b-30eb-40e1-b971-e83409fedf08-kube-api-access-5lpvl\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-cnibin\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570880    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b48ab19b-f661-4ae8-84fe-96685d29fcfb-serviceca\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-lib-modules\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-k8s-cni-cncf-io\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570951    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-netns\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570967    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-hostroot\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570981    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-host-slash\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.570997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-cni-binary-copy\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571014    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3debe752-2e41-4cd7-9265-80843637eb17-konnectivity-ca\") pod \"konnectivity-agent-7ktzp\" (UID: \"3debe752-2e41-4cd7-9265-80843637eb17\") " pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571030    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b48ab19b-f661-4ae8-84fe-96685d29fcfb-host\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-host\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-cni-binary-copy\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3debe752-2e41-4cd7-9265-80843637eb17-agent-certs\") pod \"konnectivity-agent-7ktzp\" (UID: \"3debe752-2e41-4cd7-9265-80843637eb17\") " pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysctl-d\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571110    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-os-release\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571129    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-os-release\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.571226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8763dd4f-6c77-4916-88b7-01f454d22603-dbus\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-run\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571190    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-systemd\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-var-lib-kubelet\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-kubernetes\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-socket-dir-parent\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-cni-bin\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571305    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-sys\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-cni-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571342    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-cni-multus\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571374    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-kubelet\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571402    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlcl\" (UniqueName: \"kubernetes.io/projected/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-kube-api-access-9hlcl\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571428    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-multus-certs\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571545    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-netns\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-multus-certs\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571558    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-run\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571628    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-socket-dir-parent\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571675    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8763dd4f-6c77-4916-88b7-01f454d22603-kubelet-config\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571722    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-host-slash\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572113    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-modprobe-d\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572290    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-cni-binary-copy\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572343    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-daemon-config\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572398    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-hostroot\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572468    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-cni-bin\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571629    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-kubernetes\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572535    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-kubelet\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-var-lib-cni-multus\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.572632    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-sys\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.572721    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:16:18.072695637 +0000 UTC m=+3.024039026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572790    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3debe752-2e41-4cd7-9265-80843637eb17-konnectivity-ca\") pod \"konnectivity-agent-7ktzp\" (UID: \"3debe752-2e41-4cd7-9265-80843637eb17\") " pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.572641    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572791    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-cni-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.572866 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.572855    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:18.072837804 +0000 UTC m=+3.024181188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b48ab19b-f661-4ae8-84fe-96685d29fcfb-host\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572908    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-host\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572974    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-os-release\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.572982    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysctl-d\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573005    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-cnibin\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573038    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-os-release\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-multus-conf-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573086    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8763dd4f-6c77-4916-88b7-01f454d22603-dbus\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-cni-binary-copy\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.573737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573606    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysctl-conf\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.574278 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnb2\" (UniqueName: \"kubernetes.io/projected/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-kube-api-access-dmnb2\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.574278 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.573989    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v752r\" (UniqueName: \"kubernetes.io/projected/3e0e3824-2d46-4521-a962-53e7ffce99ca-kube-api-access-v752r\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.574278 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574130    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.574278 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.571561    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-systemd\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.574564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574347    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-lib-modules\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.574564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574397    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-system-cni-dir\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.574680 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574565    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72751d1b-30eb-40e1-b971-e83409fedf08-cnibin\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.574736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574689    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-host-run-k8s-cni-cncf-io\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.574789 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574760    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72751d1b-30eb-40e1-b971-e83409fedf08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.574851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574838    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-system-cni-dir\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.574909 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574347    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b48ab19b-f661-4ae8-84fe-96685d29fcfb-serviceca\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.574963 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574895    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-var-lib-kubelet\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.575012 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574959    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-sysconfig\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.575012 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.574976    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-etc-kubernetes\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.575124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.575106    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-iptables-alerter-script\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.577079 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.577041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3debe752-2e41-4cd7-9265-80843637eb17-agent-certs\") pod \"konnectivity-agent-7ktzp\" (UID: \"3debe752-2e41-4cd7-9265-80843637eb17\") " pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.577418 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.577376    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e0e3824-2d46-4521-a962-53e7ffce99ca-tmp\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.578170 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.578143    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e0e3824-2d46-4521-a962-53e7ffce99ca-etc-tuned\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.583979 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.583954    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmq6s\" (UniqueName: \"kubernetes.io/projected/b48ab19b-f661-4ae8-84fe-96685d29fcfb-kube-api-access-lmq6s\") pod \"node-ca-nkb9d\" (UID: \"b48ab19b-f661-4ae8-84fe-96685d29fcfb\") " pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.586030 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.585971    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:17.586030 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.585997    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:17.586030 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.586011    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:17.586223 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:17.586080    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:18.086060402 +0000 UTC m=+3.037403790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:17.586977 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.586958    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlcl\" (UniqueName: \"kubernetes.io/projected/bbfb35d6-50c5-47a2-9439-b2c05127ac2c-kube-api-access-9hlcl\") pod \"iptables-alerter-gwfj6\" (UID: \"bbfb35d6-50c5-47a2-9439-b2c05127ac2c\") " pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.588351 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.588329    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpvl\" (UniqueName: \"kubernetes.io/projected/72751d1b-30eb-40e1-b971-e83409fedf08-kube-api-access-5lpvl\") pod \"multus-additional-cni-plugins-hdswt\" (UID: \"72751d1b-30eb-40e1-b971-e83409fedf08\") " pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.589384 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.589358    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v752r\" (UniqueName: \"kubernetes.io/projected/3e0e3824-2d46-4521-a962-53e7ffce99ca-kube-api-access-v752r\") pod \"tuned-2drhj\" (UID: \"3e0e3824-2d46-4521-a962-53e7ffce99ca\") " pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.591564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.591539    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rrb\" (UniqueName: \"kubernetes.io/projected/b3acd34a-68bc-472f-a6e1-8169dbed08db-kube-api-access-48rrb\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:17.591865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.591804    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnb2\" (UniqueName: \"kubernetes.io/projected/d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea-kube-api-access-dmnb2\") pod \"multus-69bq9\" (UID: \"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea\") " pod="openshift-multus/multus-69bq9"
Jul 27 23:16:17.668111 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.667827    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb"
Jul 27 23:16:17.679635 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:17.679588    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05eeaa5_6b29_4da3_8c1d_d487592f8265.slice/crio-581948122c5f78f3ba3dd5daf52e61594e893769e5a384f129b0889cf645983d WatchSource:0}: Error finding container 581948122c5f78f3ba3dd5daf52e61594e893769e5a384f129b0889cf645983d: Status 404 returned error can't find the container with id 581948122c5f78f3ba3dd5daf52e61594e893769e5a384f129b0889cf645983d
Jul 27 23:16:17.680517 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.680453    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:17.691128 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.691107    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pc986"
Jul 27 23:16:17.691573 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:17.691547    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeecc250_ce58_4af8_a233_f23355d0d0f5.slice/crio-d313bf5663d63fe2ccb1691b0d6332a05366f5879ec839761b226cfe5e8de91a WatchSource:0}: Error finding container d313bf5663d63fe2ccb1691b0d6332a05366f5879ec839761b226cfe5e8de91a: Status 404 returned error can't find the container with id d313bf5663d63fe2ccb1691b0d6332a05366f5879ec839761b226cfe5e8de91a
Jul 27 23:16:17.696443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.696410    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hdswt"
Jul 27 23:16:17.707769 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.707743    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gwfj6"
Jul 27 23:16:17.718981 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.718957    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:17.728217 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.728190    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nkb9d"
Jul 27 23:16:17.743969 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.743943    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2drhj"
Jul 27 23:16:17.755330 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:17.755308    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-69bq9"
Jul 27 23:16:18.077638 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.077607    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:18.077638 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.077644    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:18.077873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.077778    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:18.077873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.077779    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:18.077873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.077836    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:16:19.077820557 +0000 UTC m=+4.029163942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:18.077873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.077852    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:19.077844653 +0000 UTC m=+4.029188020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:18.178465 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.178419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:18.178647 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.178575    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:18.178647 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.178597    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:18.178647 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.178611    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:18.178836 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.178682    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:19.178660873 +0000 UTC m=+4.130004284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:18.397505 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.397463    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:11:17 +0000 UTC" deadline="2028-01-02 14:28:36.154295214 +0000 UTC"
Jul 27 23:16:18.397505 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.397502    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12567h12m17.756798854s"
Jul 27 23:16:18.516235 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.516202    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:18.516417 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:18.516359    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:18.528172 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.528133    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"d313bf5663d63fe2ccb1691b0d6332a05366f5879ec839761b226cfe5e8de91a"}
Jul 27 23:16:18.529777 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:18.529746    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb" event={"ID":"d05eeaa5-6b29-4da3-8c1d-d487592f8265","Type":"ContainerStarted","Data":"581948122c5f78f3ba3dd5daf52e61594e893769e5a384f129b0889cf645983d"}
Jul 27 23:16:18.885206 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:18.884612    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a32c919_8faa_44fd_9e91_d696e209a766.slice/crio-f71e87db36ebaf0705ea723a3fb3a3eefd91287d30c6d401c0472e059f5f5f7d WatchSource:0}: Error finding container f71e87db36ebaf0705ea723a3fb3a3eefd91287d30c6d401c0472e059f5f5f7d: Status 404 returned error can't find the container with id f71e87db36ebaf0705ea723a3fb3a3eefd91287d30c6d401c0472e059f5f5f7d
Jul 27 23:16:18.899271 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:18.899225    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d7cdaa_cf2d_4880_8a6d_24406bb2c7ea.slice/crio-efb5162668393559792582106a21fd9bdc93858f016626679de2f9d00dc61d56 WatchSource:0}: Error finding container efb5162668393559792582106a21fd9bdc93858f016626679de2f9d00dc61d56: Status 404 returned error can't find the container with id efb5162668393559792582106a21fd9bdc93858f016626679de2f9d00dc61d56
Jul 27 23:16:18.904616 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:18.904003    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3debe752_2e41_4cd7_9265_80843637eb17.slice/crio-566180801c2707750c7dad1b9373f9e957e934af409eef49f57eca5d788d5905 WatchSource:0}: Error finding container 566180801c2707750c7dad1b9373f9e957e934af409eef49f57eca5d788d5905: Status 404 returned error can't find the container with id 566180801c2707750c7dad1b9373f9e957e934af409eef49f57eca5d788d5905
Jul 27 23:16:18.905180 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:18.905131    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72751d1b_30eb_40e1_b971_e83409fedf08.slice/crio-86ee825e6cec6b93347bccbad760947770952668fea31c818f6ba7445116bb85 WatchSource:0}: Error finding container 86ee825e6cec6b93347bccbad760947770952668fea31c818f6ba7445116bb85: Status 404 returned error can't find the container with id 86ee825e6cec6b93347bccbad760947770952668fea31c818f6ba7445116bb85
Jul 27 23:16:18.906830 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:18.906805    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48ab19b_f661_4ae8_84fe_96685d29fcfb.slice/crio-b2843e2798f4ad3ae967b8a772cd1b0f43faefb1ed9fb27f124d87a95f09da0c WatchSource:0}: Error finding container b2843e2798f4ad3ae967b8a772cd1b0f43faefb1ed9fb27f124d87a95f09da0c: Status 404 returned error can't find the container with id b2843e2798f4ad3ae967b8a772cd1b0f43faefb1ed9fb27f124d87a95f09da0c
Jul 27 23:16:19.086380 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.086088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:19.086618 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.086437    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:19.086618 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.086361    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:19.086758 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.086637    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:21.086618006 +0000 UTC m=+6.037961398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:19.086758 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.086570    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:19.086758 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.086684    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:16:21.086672399 +0000 UTC m=+6.038015768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:19.187633 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.187032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:19.187633 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.187208    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:19.187633 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.187231    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:19.187633 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.187245    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:19.187633 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.187310    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:21.187288536 +0000 UTC m=+6.138631924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:19.398553 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.398437    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:11:17 +0000 UTC" deadline="2028-02-26 04:06:05.22820338 +0000 UTC"
Jul 27 23:16:19.398553 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.398506    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13876h49m45.829702399s"
Jul 27 23:16:19.519776 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.518928    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:19.519776 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.519103    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:19.519776 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.519195    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:19.519776 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:19.519278    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:19.542635 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.542551    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pc986" event={"ID":"4a32c919-8faa-44fd-9e91-d696e209a766","Type":"ContainerStarted","Data":"f71e87db36ebaf0705ea723a3fb3a3eefd91287d30c6d401c0472e059f5f5f7d"}
Jul 27 23:16:19.566500 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.565908    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" event={"ID":"76d1c0db1c0eb2e7cc88fd5e78b49f45","Type":"ContainerStarted","Data":"840b1ffabc8a8c6db4e7e0c5514b31ba9bc6970a442afe697795970b37e96312"}
Jul 27 23:16:19.571188 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.571095    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nkb9d" event={"ID":"b48ab19b-f661-4ae8-84fe-96685d29fcfb","Type":"ContainerStarted","Data":"b2843e2798f4ad3ae967b8a772cd1b0f43faefb1ed9fb27f124d87a95f09da0c"}
Jul 27 23:16:19.578080 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.578050    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ktzp" event={"ID":"3debe752-2e41-4cd7-9265-80843637eb17","Type":"ContainerStarted","Data":"566180801c2707750c7dad1b9373f9e957e934af409eef49f57eca5d788d5905"}
Jul 27 23:16:19.584524 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.584447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gwfj6" event={"ID":"bbfb35d6-50c5-47a2-9439-b2c05127ac2c","Type":"ContainerStarted","Data":"5aa45df13a867a9ca4934d65348045ae32f4277c73cdd3155fcc317df2e20ecd"}
Jul 27 23:16:19.587203 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.587110    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-69bq9" event={"ID":"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea","Type":"ContainerStarted","Data":"efb5162668393559792582106a21fd9bdc93858f016626679de2f9d00dc61d56"}
Jul 27 23:16:19.611637 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.611601    2570 generic.go:356] "Generic (PLEG): container finished" podID="8787e14cefbe48afd866da4ad18abf46" containerID="93b62f49ad44c2e590e28021df85d1ac27ebf8efb7b17d7d409c0a4542f45287" exitCode=0
Jul 27 23:16:19.611767 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.611700    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" event={"ID":"8787e14cefbe48afd866da4ad18abf46","Type":"ContainerDied","Data":"93b62f49ad44c2e590e28021df85d1ac27ebf8efb7b17d7d409c0a4542f45287"}
Jul 27 23:16:19.618228 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.618196    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerStarted","Data":"86ee825e6cec6b93347bccbad760947770952668fea31c818f6ba7445116bb85"}
Jul 27 23:16:19.621171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.621133    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2drhj" event={"ID":"3e0e3824-2d46-4521-a962-53e7ffce99ca","Type":"ContainerStarted","Data":"5aee3177afc6a7d624ac7da26ddba3bd6f32e713843d4ec8998c0269a3d1af9f"}
Jul 27 23:16:19.632965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:19.632789    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" podStartSLOduration=2.632768731 podStartE2EDuration="2.632768731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:19.583522647 +0000 UTC m=+4.534866041" watchObservedRunningTime="2026-07-27 23:16:19.632768731 +0000 UTC m=+4.584112123"
Jul 27 23:16:20.517769 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:20.517244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:20.517769 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:20.517404    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:20.637342 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:20.636945    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" event={"ID":"8787e14cefbe48afd866da4ad18abf46","Type":"ContainerStarted","Data":"ead6499c8cd9a4b30c243097947d4d7c6db7f085c53cc564d2cea5688295e111"}
Jul 27 23:16:20.654160 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:20.653337    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" podStartSLOduration=3.653313307 podStartE2EDuration="3.653313307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:20.652657921 +0000 UTC m=+5.604001312" watchObservedRunningTime="2026-07-27 23:16:20.653313307 +0000 UTC m=+5.604656698"
Jul 27 23:16:21.108517 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:21.107832    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:21.108517 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:21.107886    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:21.108517 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.108036    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:21.108517 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.108093    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:16:25.108076772 +0000 UTC m=+10.059420139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:21.108517 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.108440    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:21.108517 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.108489    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:25.108474539 +0000 UTC m=+10.059817909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:21.209506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:21.208721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:21.209506 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.208966    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:21.209506 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.208992    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:21.209506 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.209006    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:21.209506 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.209082    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:25.209061841 +0000 UTC m=+10.160405213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:21.521803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:21.521769    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:21.522271 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.521969    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:21.522552 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:21.522526    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:21.522712 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:21.522683    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:22.516693 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:22.516654    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:22.516881 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:22.516806    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:23.517510 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:23.517471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:23.517510 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:23.517512    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:23.517922 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:23.517617    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:23.517922 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:23.517754    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:24.516158 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:24.516129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:24.516302 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:24.516243    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:25.144603 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:25.144557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:25.145086 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:25.144615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:25.145086 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.144729    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:25.145086 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.144726    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:25.145086 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.144782    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:16:33.14476724 +0000 UTC m=+18.096110606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:25.145086 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.144806    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:33.144788433 +0000 UTC m=+18.096131822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:25.245864 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:25.245820    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:25.246023 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.246010    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:25.246063 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.246030    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:25.246063 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.246042    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:25.246118 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.246100    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:33.246081918 +0000 UTC m=+18.197425290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:25.517563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:25.517481    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:25.517563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:25.517528    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:25.517758 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.517640    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:25.517758 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:25.517736    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:26.516503 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:26.516467    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:26.516884 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:26.516605    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:27.516003 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:27.515978    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:27.516174 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:27.515979    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:27.516222 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:27.516073    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:27.516264 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:27.516241    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:28.516256 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:28.516215    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:28.516782 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:28.516380    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:29.517155 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:29.516608    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:29.517155 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:29.516769    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:29.517155 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:29.516825    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:29.517155 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:29.516944    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:30.516712 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:30.516674    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:30.516892 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:30.516793    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:31.516369 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:31.516314    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:31.516772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:31.516384    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:31.516772 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:31.516474    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:31.516772 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:31.516591    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:32.516261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:32.516223    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:32.516500 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:32.516365    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:33.203370 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:33.203317    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:33.203562 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:33.203377    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:33.203562 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.203470    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:33.203562 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.203510    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:33.203562 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.203538    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:49.203522634 +0000 UTC m=+34.154866019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:33.203562 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.203561    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:16:49.203545531 +0000 UTC m=+34.154888896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:33.304772 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:33.304729    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:33.304946 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.304924    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:33.305008 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.304952    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:33.305008 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.304964    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:33.305101 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.305023    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:49.305003088 +0000 UTC m=+34.256346456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:33.515987 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:33.515907    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:33.516146 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.516037    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:33.516428 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:33.516411    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:33.516550 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:33.516529    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:34.516261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:34.516220    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:34.516522 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:34.516388    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:35.517729 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:35.517221    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:35.517729 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:35.517365    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:35.517729 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:35.517410    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:35.517729 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:35.517487    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:36.515940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:36.515903    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:36.516126 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:36.516044    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:37.516261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:37.516226    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:37.516966 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:37.516382    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:37.516966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:37.516226    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:37.516966 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:37.516874    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:38.516924 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.516608    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:38.517898 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:38.517032    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:38.674643 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.674603    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2drhj" event={"ID":"3e0e3824-2d46-4521-a962-53e7ffce99ca","Type":"ContainerStarted","Data":"cc7f632044ce7548261cc89b12e7252787a6a5bccfa3d303b98166aa6f31f44e"}
Jul 27 23:16:38.676345 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.676316    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pc986" event={"ID":"4a32c919-8faa-44fd-9e91-d696e209a766","Type":"ContainerStarted","Data":"9c04aae45dcfcc5a6c175c581a87b56f9cace8e689698ba887b3f87ff5540bca"}
Jul 27 23:16:38.679653 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.679630    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"40794b2865b7872be2e047a4690768974f3d059fb7e02f6281828f62dbd343b7"}
Jul 27 23:16:38.679777 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.679712    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"a7c53708e56a29ce365c66c4c0e6e6f42d04409fb0f9952af8ec8cbf6320cb28"}
Jul 27 23:16:38.679777 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.679748    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"d639ac53d644c062755ce390311aa4d5148442ed65c9a793fd3983a977470637"}
Jul 27 23:16:38.679882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.679789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"b9c70e5edd72d8838bf258c9f1b32201de93140d843e79e44789e0f279257028"}
Jul 27 23:16:38.679882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.679832    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"798e2e859f6f540fba41db17407dff4bce844830a22c926228d1922ffd730872"}
Jul 27 23:16:38.679882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.679879    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"272245968c8ef4315e8529cd5251aaceffaee3efb88576d46decd5910dabeae4"}
Jul 27 23:16:38.681205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.681183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb" event={"ID":"d05eeaa5-6b29-4da3-8c1d-d487592f8265","Type":"ContainerStarted","Data":"84ffc8a8dc1b07a0f3d636bef7f76dfb2db8746aac8721fd4cadd854e8da7a40"}
Jul 27 23:16:38.682708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.682684    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nkb9d" event={"ID":"b48ab19b-f661-4ae8-84fe-96685d29fcfb","Type":"ContainerStarted","Data":"b9749e904d832e609c30d69c3a4d0e69f2aad3d395deb6fb877ff22bc013284c"}
Jul 27 23:16:38.684266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.684215    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7ktzp" event={"ID":"3debe752-2e41-4cd7-9265-80843637eb17","Type":"ContainerStarted","Data":"dafc391684a4b8524d53af5ae51bfb8aeb70855bd4f92198c008302e810fb8c9"}
Jul 27 23:16:38.685873 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.685849    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-69bq9" event={"ID":"d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea","Type":"ContainerStarted","Data":"357dd8a707c311449494130a786f81f6b34bec4b61cbd7a19ab9fa96d9afd7a0"}
Jul 27 23:16:38.687509 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.687487    2570 generic.go:356] "Generic (PLEG): container finished" podID="72751d1b-30eb-40e1-b971-e83409fedf08" containerID="a2b216a55c48f45bf3a1c75155158baf990a803c3d5b5d1b2623cfcfe7556a1b" exitCode=0
Jul 27 23:16:38.687509 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.687517    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerDied","Data":"a2b216a55c48f45bf3a1c75155158baf990a803c3d5b5d1b2623cfcfe7556a1b"}
Jul 27 23:16:38.729681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.729632    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2drhj" podStartSLOduration=3.936967793 podStartE2EDuration="22.729615537s" totalImagesPullingTime="18.792647744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:38.713131873 +0000 UTC m=+23.664475281" watchObservedRunningTime="2026-07-27 23:16:38.729615537 +0000 UTC m=+23.680958934"
Jul 27 23:16:38.729866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.729762    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pc986" podStartSLOduration=3.889800967 podStartE2EDuration="22.72975605s" totalImagesPullingTime="18.839955083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:38.729610104 +0000 UTC m=+23.680953490" watchObservedRunningTime="2026-07-27 23:16:38.72975605 +0000 UTC m=+23.681099436"
Jul 27 23:16:38.767990 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.767924    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nkb9d" podStartSLOduration=3.977706876 podStartE2EDuration="22.767902798s" totalImagesPullingTime="18.790195922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:38.767172768 +0000 UTC m=+23.718516182" watchObservedRunningTime="2026-07-27 23:16:38.767902798 +0000 UTC m=+23.719246196"
Jul 27 23:16:38.784890 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.784837    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-69bq9" podStartSLOduration=3.885493006 podStartE2EDuration="22.784818405s" totalImagesPullingTime="18.899325399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:38.784379954 +0000 UTC m=+23.735723334" watchObservedRunningTime="2026-07-27 23:16:38.784818405 +0000 UTC m=+23.736161792"
Jul 27 23:16:38.797929 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:38.797885    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7ktzp" podStartSLOduration=4.008585933 podStartE2EDuration="22.797870488s" totalImagesPullingTime="18.789284555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:38.797757779 +0000 UTC m=+23.749101166" watchObservedRunningTime="2026-07-27 23:16:38.797870488 +0000 UTC m=+23.749213874"
Jul 27 23:16:39.219699 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.219670    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 23:16:39.425545 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.425431    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T23:16:39.219695775Z","UUID":"2ac8c6c9-3d0e-4bf5-99b2-b962fa0e956e","Handler":null,"Name":"","Endpoint":""}
Jul 27 23:16:39.427258 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.427236    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 23:16:39.427411 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.427270    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 23:16:39.516095 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.516065    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:39.516095 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.516081    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:39.516373 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:39.516200    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:39.516373 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:39.516319    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:39.691579 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.691500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb" event={"ID":"d05eeaa5-6b29-4da3-8c1d-d487592f8265","Type":"ContainerStarted","Data":"2d15c199edf4bf45ce3a5f5d4d30caf2b4e288f0280b66e19f245dc8c2d757aa"}
Jul 27 23:16:39.693294 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.693258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gwfj6" event={"ID":"bbfb35d6-50c5-47a2-9439-b2c05127ac2c","Type":"ContainerStarted","Data":"187a733eccda9ccb9705361281ce3fed16551df5bfae13e52223d985ff89d12e"}
Jul 27 23:16:39.708126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:39.708084    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gwfj6" podStartSLOduration=4.87250201 podStartE2EDuration="23.708068025s" totalImagesPullingTime="18.835566015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:39.70775227 +0000 UTC m=+24.659095657" watchObservedRunningTime="2026-07-27 23:16:39.708068025 +0000 UTC m=+24.659411412"
Jul 27 23:16:40.372718 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:40.372689    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:40.373532 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:40.373504    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:40.516488 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:40.516450    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:40.516666 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:40.516586    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:40.699672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:40.699634    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"ce8c59a4046c4a5d9597bc9fe541ec500ca25248fae04b2e61249d8c85bed041"}
Jul 27 23:16:40.702259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:40.702225    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb" event={"ID":"d05eeaa5-6b29-4da3-8c1d-d487592f8265","Type":"ContainerStarted","Data":"4afc58108a8f0534aa0961820e9f47c2a15be0a987d117cf34d5907860e63e34"}
Jul 27 23:16:40.718630 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:40.718570    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6tnpb" podStartSLOduration=2.447637894 podStartE2EDuration="24.718547793s" totalImagesPullingTime="22.270909899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:40.718316409 +0000 UTC m=+25.669659824" watchObservedRunningTime="2026-07-27 23:16:40.718547793 +0000 UTC m=+25.669891186"
Jul 27 23:16:41.515932 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:41.515894    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:41.516133 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:41.515899    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:41.516133 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:41.516027    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:41.516218 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:41.516146    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:41.704893 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:41.704861    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 23:16:42.516760 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:42.516583    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:42.516897 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:42.516853    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:43.517006 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.516773    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:43.517773 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.516848    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:43.517773 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:43.517054    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:43.517773 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:43.517123    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:43.712216 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.712175    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" event={"ID":"beecc250-ce58-4af8-a233-f23355d0d0f5","Type":"ContainerStarted","Data":"22b0588d75439166ffb2dbce24f52fdb2892cf8adf426e045e47d2cc628eddba"}
Jul 27 23:16:43.712444 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.712426    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:43.712516 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.712495    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:43.714090 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.714062    2570 generic.go:356] "Generic (PLEG): container finished" podID="72751d1b-30eb-40e1-b971-e83409fedf08" containerID="0547fe076f1f561b339bc13a42ba6512e5e9d82b72b49445113075edeff8a122" exitCode=0
Jul 27 23:16:43.714218 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.714098    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerDied","Data":"0547fe076f1f561b339bc13a42ba6512e5e9d82b72b49445113075edeff8a122"}
Jul 27 23:16:43.729582 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.729558    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:43.729711 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.729692    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:43.739132 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:43.739090    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg" podStartSLOduration=7.622657964 podStartE2EDuration="27.73907586s" totalImagesPullingTime="20.116417896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:43.737399843 +0000 UTC m=+28.688743245" watchObservedRunningTime="2026-07-27 23:16:43.73907586 +0000 UTC m=+28.690419247"
Jul 27 23:16:44.516548 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:44.516510    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:44.516856 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:44.516630    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:44.716474 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:44.716445    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:16:45.181444 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.181415    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd7ss"]
Jul 27 23:16:45.181623 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.181575    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:45.181681 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:45.181660    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:45.184107 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.184084    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hvt44"]
Jul 27 23:16:45.184234 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.184171    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:45.184290 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:45.184268    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:45.184640 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.184616    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vbgm4"]
Jul 27 23:16:45.184780 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.184739    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:45.184866 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:45.184839    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:45.720241 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.720204    2570 generic.go:356] "Generic (PLEG): container finished" podID="72751d1b-30eb-40e1-b971-e83409fedf08" containerID="c0fd0ee0f3b12479af253990ad49c2862162d631d2c17ab7c4388255097c60e5" exitCode=0
Jul 27 23:16:45.720687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:45.720294    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerDied","Data":"c0fd0ee0f3b12479af253990ad49c2862162d631d2c17ab7c4388255097c60e5"}
Jul 27 23:16:46.516827 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:46.516797    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:46.516985 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:46.516797    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:46.516985 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:46.516905    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:46.517066 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:46.517049    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:47.020089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:47.019859    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:47.020503 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:47.020238    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 23:16:47.020978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:47.020733    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7ktzp"
Jul 27 23:16:47.516664 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:47.516637    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:47.516888 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:47.516736    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:47.727171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:47.727130    2570 generic.go:356] "Generic (PLEG): container finished" podID="72751d1b-30eb-40e1-b971-e83409fedf08" containerID="cf90cfcc6a5259bf9228b554b0df4c4b4c8408a426f5f11098beb076e5f6a53b" exitCode=0
Jul 27 23:16:47.727342 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:47.727193    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerDied","Data":"cf90cfcc6a5259bf9228b554b0df4c4b4c8408a426f5f11098beb076e5f6a53b"}
Jul 27 23:16:48.516530 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:48.516495    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:48.516976 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:48.516499    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:48.516976 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:48.516607    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:48.516976 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:48.516703    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:49.227619 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:49.227586    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:49.227619 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:49.227634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:49.227873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.227753    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:49.227873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.227772    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:49.227873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.227832    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:17:21.227813881 +0000 UTC m=+66.179157247 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 23:16:49.227873 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.227851    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret podName:8763dd4f-6c77-4916-88b7-01f454d22603 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:21.22784178 +0000 UTC m=+66.179185151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret") pod "global-pull-secret-syncer-hvt44" (UID: "8763dd4f-6c77-4916-88b7-01f454d22603") : object "kube-system"/"original-pull-secret" not registered
Jul 27 23:16:49.328948 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:49.328914    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:49.329148 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.329054    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 23:16:49.329148 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.329070    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 23:16:49.329148 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.329080    2570 projected.go:196] Error preparing data for projected volume kube-api-access-qrc4d for pod openshift-network-diagnostics/network-check-target-cd7ss: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:49.329148 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.329144    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d podName:ee6d39b3-1018-4e6d-8f92-5d7b81620807 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:21.329124682 +0000 UTC m=+66.280468068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrc4d" (UniqueName: "kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d") pod "network-check-target-cd7ss" (UID: "ee6d39b3-1018-4e6d-8f92-5d7b81620807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 23:16:49.516328 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:49.516250    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:49.516507 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:49.516414    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cd7ss" podUID="ee6d39b3-1018-4e6d-8f92-5d7b81620807"
Jul 27 23:16:50.516391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:50.516343    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:50.516970 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:50.516491    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hvt44" podUID="8763dd4f-6c77-4916-88b7-01f454d22603"
Jul 27 23:16:50.516970 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:50.516551    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:50.516970 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:50.516666    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:16:51.410306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.408607    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeReady"
Jul 27 23:16:51.410306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.408773    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 23:16:51.449617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.449585    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zw7cg"]
Jul 27 23:16:51.453609 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.453562    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kbc6r"]
Jul 27 23:16:51.453749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.453704    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.456449 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.456420    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 23:16:51.456576 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.456566    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 23:16:51.456671 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.456632    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qdw\""
Jul 27 23:16:51.457197 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.457178    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.459454 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.459421    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 23:16:51.459940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.459922    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 23:16:51.460098 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.459983    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-txnj2\""
Jul 27 23:16:51.460098 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.459923    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 23:16:51.461766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.461741    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zw7cg"]
Jul 27 23:16:51.464552 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.464533    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kbc6r"]
Jul 27 23:16:51.520381 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.520288    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:16:51.523563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.523539    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 23:16:51.523687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.523600    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ft576\""
Jul 27 23:16:51.523687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.523539    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 23:16:51.542655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.542630    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zpd\" (UniqueName: \"kubernetes.io/projected/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-kube-api-access-82zpd\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.542793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.542662    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkrw\" (UniqueName: \"kubernetes.io/projected/d5fb248c-397e-4373-9e50-8c22438a3fb1-kube-api-access-qmkrw\") pod \"ingress-canary-kbc6r\" (UID: \"d5fb248c-397e-4373-9e50-8c22438a3fb1\") " pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.542793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.542697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5fb248c-397e-4373-9e50-8c22438a3fb1-cert\") pod \"ingress-canary-kbc6r\" (UID: \"d5fb248c-397e-4373-9e50-8c22438a3fb1\") " pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.542793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.542760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-config-volume\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.542793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.542783    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-tmp-dir\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.542958 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.542802    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.644018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.643968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-config-volume\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.644018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-tmp-dir\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.644261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.644261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zpd\" (UniqueName: \"kubernetes.io/projected/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-kube-api-access-82zpd\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.644261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkrw\" (UniqueName: \"kubernetes.io/projected/d5fb248c-397e-4373-9e50-8c22438a3fb1-kube-api-access-qmkrw\") pod \"ingress-canary-kbc6r\" (UID: \"d5fb248c-397e-4373-9e50-8c22438a3fb1\") " pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.644261 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:51.644220    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:51.644261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5fb248c-397e-4373-9e50-8c22438a3fb1-cert\") pod \"ingress-canary-kbc6r\" (UID: \"d5fb248c-397e-4373-9e50-8c22438a3fb1\") " pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.644544 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:51.644289    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:16:52.144268458 +0000 UTC m=+37.095611841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:51.644544 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644348    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-tmp-dir\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.644687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.644661    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-config-volume\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.648631 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.648609    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5fb248c-397e-4373-9e50-8c22438a3fb1-cert\") pod \"ingress-canary-kbc6r\" (UID: \"d5fb248c-397e-4373-9e50-8c22438a3fb1\") " pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.656052 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.654166    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkrw\" (UniqueName: \"kubernetes.io/projected/d5fb248c-397e-4373-9e50-8c22438a3fb1-kube-api-access-qmkrw\") pod \"ingress-canary-kbc6r\" (UID: \"d5fb248c-397e-4373-9e50-8c22438a3fb1\") " pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.656052 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.654319    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zpd\" (UniqueName: \"kubernetes.io/projected/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-kube-api-access-82zpd\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:51.740528 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.740499    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerStarted","Data":"529df84c34bad8814bfc2d5ac2ca52627c317e1bebf773d46fba3bfd764a83f5"}
Jul 27 23:16:51.773902 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.773832    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kbc6r"
Jul 27 23:16:51.913440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:51.913186    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kbc6r"]
Jul 27 23:16:51.915988 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:16:51.915948    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fb248c_397e_4373_9e50_8c22438a3fb1.slice/crio-39fec479ee44b8dfc2ead6ea468649084b9821b90ba773afd44817df10f80903 WatchSource:0}: Error finding container 39fec479ee44b8dfc2ead6ea468649084b9821b90ba773afd44817df10f80903: Status 404 returned error can't find the container with id 39fec479ee44b8dfc2ead6ea468649084b9821b90ba773afd44817df10f80903
Jul 27 23:16:52.149043 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.149010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:52.149210 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:52.149170    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:52.149255 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:52.149229    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:16:53.149215078 +0000 UTC m=+38.100558443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:52.516077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.515993    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:16:52.516077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.516042    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:16:52.518981 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.518953    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n98qx\""
Jul 27 23:16:52.519102 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.518983    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 23:16:52.519292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.519267    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 23:16:52.745444 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.745409    2570 generic.go:356] "Generic (PLEG): container finished" podID="72751d1b-30eb-40e1-b971-e83409fedf08" containerID="529df84c34bad8814bfc2d5ac2ca52627c317e1bebf773d46fba3bfd764a83f5" exitCode=0
Jul 27 23:16:52.746086 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.745494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerDied","Data":"529df84c34bad8814bfc2d5ac2ca52627c317e1bebf773d46fba3bfd764a83f5"}
Jul 27 23:16:52.746685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:52.746611    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kbc6r" event={"ID":"d5fb248c-397e-4373-9e50-8c22438a3fb1","Type":"ContainerStarted","Data":"39fec479ee44b8dfc2ead6ea468649084b9821b90ba773afd44817df10f80903"}
Jul 27 23:16:53.157485 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:53.157455    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:53.157653 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:53.157570    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:53.157653 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:53.157625    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:16:55.157612095 +0000 UTC m=+40.108955460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:53.752217 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:53.752188    2570 generic.go:356] "Generic (PLEG): container finished" podID="72751d1b-30eb-40e1-b971-e83409fedf08" containerID="2c064c12aa787269d00cb3fabea63e852a682ff5783d2b052ca4a00319fc6e85" exitCode=0
Jul 27 23:16:53.752675 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:53.752244    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerDied","Data":"2c064c12aa787269d00cb3fabea63e852a682ff5783d2b052ca4a00319fc6e85"}
Jul 27 23:16:54.758102 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:54.757919    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hdswt" event={"ID":"72751d1b-30eb-40e1-b971-e83409fedf08","Type":"ContainerStarted","Data":"68df53befc494e72612e5c9ce2fd73fc673672de24458067d20acac3da65dd86"}
Jul 27 23:16:55.174491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:55.174442    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:55.174655 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:55.174592    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:55.174736 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:55.174656    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:16:59.174639283 +0000 UTC m=+44.125982648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:16:55.761392 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:55.761333    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kbc6r" event={"ID":"d5fb248c-397e-4373-9e50-8c22438a3fb1","Type":"ContainerStarted","Data":"33ac6ee41845ea05ad0ba126dc4612c07f3f321b317e023f29ee414d4173a598"}
Jul 27 23:16:55.777063 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:55.777015    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kbc6r" podStartSLOduration=1.850779805 podStartE2EDuration="4.776999633s" totalImagesPullingTime="2.926219828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:55.776537816 +0000 UTC m=+40.727881202" watchObservedRunningTime="2026-07-27 23:16:55.776999633 +0000 UTC m=+40.728343022"
Jul 27 23:16:55.777212 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:55.777086    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hdswt" podStartSLOduration=10.371881097 podStartE2EDuration="39.777082225s" totalImagesPullingTime="29.405201128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:54.789633048 +0000 UTC m=+39.740976428" watchObservedRunningTime="2026-07-27 23:16:55.777082225 +0000 UTC m=+40.728425612"
Jul 27 23:16:59.203752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:16:59.203710    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:16:59.204143 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:59.203865    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:16:59.204143 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:16:59.203936    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:17:07.203920678 +0000 UTC m=+52.155264046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:07.263402 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:07.263334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:17:07.263886 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:07.263497    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:07.263886 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:07.263567    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:17:23.263548125 +0000 UTC m=+68.214891493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:15.731811 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:15.731771    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gpjmg"
Jul 27 23:17:21.261077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.261031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:17:21.261530 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.261091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:17:21.264238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.264222    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 23:17:21.264411 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.264394    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 23:17:21.271557 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:21.271542    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 23:17:21.271632 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:21.271598    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:18:25.271581752 +0000 UTC m=+130.222925117 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : secret "metrics-daemon-secret" not found
Jul 27 23:17:21.275312 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.275289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8763dd4f-6c77-4916-88b7-01f454d22603-original-pull-secret\") pod \"global-pull-secret-syncer-hvt44\" (UID: \"8763dd4f-6c77-4916-88b7-01f454d22603\") " pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:17:21.328541 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.328501    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hvt44"
Jul 27 23:17:21.361563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.361531    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:17:21.364499 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.364481    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 23:17:21.375232 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.375203    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 23:17:21.385430 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.385411    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrc4d\" (UniqueName: \"kubernetes.io/projected/ee6d39b3-1018-4e6d-8f92-5d7b81620807-kube-api-access-qrc4d\") pod \"network-check-target-cd7ss\" (UID: \"ee6d39b3-1018-4e6d-8f92-5d7b81620807\") " pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:17:21.454924 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.454896    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hvt44"]
Jul 27 23:17:21.458321 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:17:21.458295    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8763dd4f_6c77_4916_88b7_01f454d22603.slice/crio-92d98126cdcd46850304240a739a503b820a6759d78db3a43fcd0f64fdfe5ec6 WatchSource:0}: Error finding container 92d98126cdcd46850304240a739a503b820a6759d78db3a43fcd0f64fdfe5ec6: Status 404 returned error can't find the container with id 92d98126cdcd46850304240a739a503b820a6759d78db3a43fcd0f64fdfe5ec6
Jul 27 23:17:21.532695 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.532620    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ft576\""
Jul 27 23:17:21.540783 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.540763    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:17:21.661585 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.661068    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cd7ss"]
Jul 27 23:17:21.823258 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.823162    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd7ss" event={"ID":"ee6d39b3-1018-4e6d-8f92-5d7b81620807","Type":"ContainerStarted","Data":"287e142e7d3804bc7b43afb9a8e7ceb3e47ee3e36cc668733b1c633d997ac3b7"}
Jul 27 23:17:21.824241 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:21.824218    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hvt44" event={"ID":"8763dd4f-6c77-4916-88b7-01f454d22603","Type":"ContainerStarted","Data":"92d98126cdcd46850304240a739a503b820a6759d78db3a43fcd0f64fdfe5ec6"}
Jul 27 23:17:23.274508 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:23.274460    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:17:23.274907 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:23.274667    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:23.274907 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:23.274749    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:17:55.274726807 +0000 UTC m=+100.226070175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:26.839856 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:26.839821    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cd7ss" event={"ID":"ee6d39b3-1018-4e6d-8f92-5d7b81620807","Type":"ContainerStarted","Data":"70edc69c34c55d5a47ff7de42f223df86549c66071629d6d7a6c869745d98e4b"}
Jul 27 23:17:26.840268 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:26.839893    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:17:26.841253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:26.841231    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hvt44" event={"ID":"8763dd4f-6c77-4916-88b7-01f454d22603","Type":"ContainerStarted","Data":"f13d4ce919be40cebb1f2e78e9f9544c1826a580b915303ce84556815ba6e8f3"}
Jul 27 23:17:26.856558 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:26.856514    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cd7ss" podStartSLOduration=66.274082109 podStartE2EDuration="1m10.8565019s" totalImagesPullingTime="4.582419791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:26.855531674 +0000 UTC m=+71.806875063" watchObservedRunningTime="2026-07-27 23:17:26.8565019 +0000 UTC m=+71.807845264"
Jul 27 23:17:26.869322 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:26.869280    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hvt44" podStartSLOduration=66.059032688 podStartE2EDuration="1m10.869266855s" totalImagesPullingTime="4.810234167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:26.869077944 +0000 UTC m=+71.820421351" watchObservedRunningTime="2026-07-27 23:17:26.869266855 +0000 UTC m=+71.820610242"
Jul 27 23:17:43.400042 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.399927    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"]
Jul 27 23:17:43.403803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.403788    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.407780 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.407755    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 23:17:43.407950 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.407829    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 23:17:43.407950 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.407843    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-44v2c\""
Jul 27 23:17:43.408113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.407843    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 23:17:43.408113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.407855    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 23:17:43.411961 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.411939    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"]
Jul 27 23:17:43.518161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.518124    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npm59\" (UniqueName: \"kubernetes.io/projected/b9382444-a8f9-413f-8784-d843b176763f-kube-api-access-npm59\") pod \"managed-serviceaccount-addon-agent-66c58885f4-nvwtm\" (UID: \"b9382444-a8f9-413f-8784-d843b176763f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.518326 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.518176    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9382444-a8f9-413f-8784-d843b176763f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66c58885f4-nvwtm\" (UID: \"b9382444-a8f9-413f-8784-d843b176763f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.618968 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.618931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npm59\" (UniqueName: \"kubernetes.io/projected/b9382444-a8f9-413f-8784-d843b176763f-kube-api-access-npm59\") pod \"managed-serviceaccount-addon-agent-66c58885f4-nvwtm\" (UID: \"b9382444-a8f9-413f-8784-d843b176763f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.619145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.618974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9382444-a8f9-413f-8784-d843b176763f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66c58885f4-nvwtm\" (UID: \"b9382444-a8f9-413f-8784-d843b176763f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.621770 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.621745    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9382444-a8f9-413f-8784-d843b176763f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-66c58885f4-nvwtm\" (UID: \"b9382444-a8f9-413f-8784-d843b176763f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.626568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.626544    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npm59\" (UniqueName: \"kubernetes.io/projected/b9382444-a8f9-413f-8784-d843b176763f-kube-api-access-npm59\") pod \"managed-serviceaccount-addon-agent-66c58885f4-nvwtm\" (UID: \"b9382444-a8f9-413f-8784-d843b176763f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.721492 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.721406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"
Jul 27 23:17:43.842869 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.842826    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm"]
Jul 27 23:17:43.846608 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:17:43.846574    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9382444_a8f9_413f_8784_d843b176763f.slice/crio-97856f163cb42bded62f6c020282e5ebc46bdef54014be86d37d3bcf21272b3f WatchSource:0}: Error finding container 97856f163cb42bded62f6c020282e5ebc46bdef54014be86d37d3bcf21272b3f: Status 404 returned error can't find the container with id 97856f163cb42bded62f6c020282e5ebc46bdef54014be86d37d3bcf21272b3f
Jul 27 23:17:43.883187 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:43.883153    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm" event={"ID":"b9382444-a8f9-413f-8784-d843b176763f","Type":"ContainerStarted","Data":"97856f163cb42bded62f6c020282e5ebc46bdef54014be86d37d3bcf21272b3f"}
Jul 27 23:17:46.892001 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:46.891971    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm" event={"ID":"b9382444-a8f9-413f-8784-d843b176763f","Type":"ContainerStarted","Data":"276401ee0ca1448ca13826814979e2f56ca2ba90cec9fccefe5828f213cc5c83"}
Jul 27 23:17:46.906963 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:46.906908    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm" podStartSLOduration=1.220920897 podStartE2EDuration="3.90689433s" totalImagesPullingTime="2.685973433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:46.906387518 +0000 UTC m=+91.857730924" watchObservedRunningTime="2026-07-27 23:17:46.90689433 +0000 UTC m=+91.858237742"
Jul 27 23:17:55.298609 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:55.298566    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:17:55.298999 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:55.298674    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:17:55.298999 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:17:55.298726    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:18:59.298713325 +0000 UTC m=+164.250056691 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:17:57.846064 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:17:57.846028    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cd7ss"
Jul 27 23:18:25.298231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:18:25.298175    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:18:25.298763 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:18:25.298328    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 23:18:25.298763 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:18:25.298418    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs podName:b3acd34a-68bc-472f-a6e1-8169dbed08db nodeName:}" failed. No retries permitted until 2026-07-27 23:20:27.29840026 +0000 UTC m=+252.249743625 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs") pod "network-metrics-daemon-vbgm4" (UID: "b3acd34a-68bc-472f-a6e1-8169dbed08db") : secret "metrics-daemon-secret" not found
Jul 27 23:18:54.467324 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:18:54.467281    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zw7cg" podUID="fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c"
Jul 27 23:18:55.050622 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:18:55.050588    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:18:55.533905 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:18:55.533869    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vbgm4" podUID="b3acd34a-68bc-472f-a6e1-8169dbed08db"
Jul 27 23:18:59.339655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:18:59.339619    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:18:59.340105 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:18:59.339736    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 23:18:59.340105 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:18:59.339789    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls podName:fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c nodeName:}" failed. No retries permitted until 2026-07-27 23:21:01.339774954 +0000 UTC m=+286.291118320 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls") pod "dns-default-zw7cg" (UID: "fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c") : secret "dns-default-metrics-tls" not found
Jul 27 23:19:06.516405 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:06.516270    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:19:47.178577 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:47.178544    2570 generic.go:356] "Generic (PLEG): container finished" podID="b9382444-a8f9-413f-8784-d843b176763f" containerID="276401ee0ca1448ca13826814979e2f56ca2ba90cec9fccefe5828f213cc5c83" exitCode=255
Jul 27 23:19:47.179058 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:47.178602    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm" event={"ID":"b9382444-a8f9-413f-8784-d843b176763f","Type":"ContainerDied","Data":"276401ee0ca1448ca13826814979e2f56ca2ba90cec9fccefe5828f213cc5c83"}
Jul 27 23:19:47.184199 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:47.184178    2570 scope.go:117] "RemoveContainer" containerID="276401ee0ca1448ca13826814979e2f56ca2ba90cec9fccefe5828f213cc5c83"
Jul 27 23:19:48.182728 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:48.182688    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-66c58885f4-nvwtm" event={"ID":"b9382444-a8f9-413f-8784-d843b176763f","Type":"ContainerStarted","Data":"435e672ef6751b22e3efaae79b10c3cc5c170c3e3da814807f68efe3b241882c"}
Jul 27 23:19:54.707114 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.707082    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tcw4n"]
Jul 27 23:19:54.710236 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.710219    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-867c9589c8-fgvqc"]
Jul 27 23:19:54.710385 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.710368    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.714156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.713902    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 23:19:54.714404 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.714387    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 23:19:54.715538 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.715511    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.715693 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.715671    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 23:19:54.715999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.715976    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xwtbj\""
Jul 27 23:19:54.719087 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.718983    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:19:54.719377 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.719334    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 23:19:54.719920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.719774    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-66lfc\""
Jul 27 23:19:54.719920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.719779    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 23:19:54.719920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.719841    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 23:19:54.720276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.720260    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 23:19:54.726124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.726103    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 23:19:54.727129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.727110    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tcw4n"]
Jul 27 23:19:54.730467 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.730448    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-867c9589c8-fgvqc"]
Jul 27 23:19:54.805578 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805537    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"]
Jul 27 23:19:54.805751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.805751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-installation-pull-secrets\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.805830 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805760    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-config\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.805830 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-trusted-ca\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.805830 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805821    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-certificates\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.805948 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805848    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-trusted-ca\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.805948 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805923    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-image-registry-private-configuration\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.806019 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.805972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-serving-cert\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.806019 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.806008    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-bound-sa-token\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.806094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.806027    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9f7j\" (UniqueName: \"kubernetes.io/projected/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-kube-api-access-s9f7j\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.806094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.806082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd51a6d-b85c-470a-9933-73fef8e40628-ca-trust-extracted\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.806192 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.806128    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-kube-api-access-wzrjn\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.808667 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.808652    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:54.811943 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.811914    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 27 23:19:54.812069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.811946    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 27 23:19:54.812069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.811946    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8p5d2\""
Jul 27 23:19:54.812178 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.812136    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:19:54.817975 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.817944    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"]
Jul 27 23:19:54.901646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.901611    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"]
Jul 27 23:19:54.904468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.904451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:54.906844 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906846    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-installation-pull-secrets\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906878    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-config\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-trusted-ca\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-certificates\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:54.906935    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:54.906951    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867c9589c8-fgvqc: secret "image-registry-tls" not found
Jul 27 23:19:54.906965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906959    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-trusted-ca\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.907299 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.906986    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:54.907299 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:54.907005    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls podName:8cd51a6d-b85c-470a-9933-73fef8e40628 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:55.406984858 +0000 UTC m=+220.358328236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls") pod "image-registry-867c9589c8-fgvqc" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628") : secret "image-registry-tls" not found
Jul 27 23:19:54.907299 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907285    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-image-registry-private-configuration\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.907485 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-serving-cert\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.907485 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907379    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-bound-sa-token\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.907485 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907412    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9f7j\" (UniqueName: \"kubernetes.io/projected/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-kube-api-access-s9f7j\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.907485 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907458    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd51a6d-b85c-470a-9933-73fef8e40628-ca-trust-extracted\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.907672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907513    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-kube-api-access-wzrjn\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.907672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907558    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2z2\" (UniqueName: \"kubernetes.io/projected/a953c104-e172-4e38-8821-71efcc12d4e0-kube-api-access-xd2z2\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:54.907672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907644    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-certificates\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.908007 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.907968    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 23:19:54.908405 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.908249    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-config\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.908405 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.908265    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd51a6d-b85c-470a-9933-73fef8e40628-ca-trust-extracted\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.908760 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.908742    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 23:19:54.909010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.908742    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:19:54.909133 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.909096    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 23:19:54.909445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.909241    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-trusted-ca\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.909445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.909040    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r6l4x\""
Jul 27 23:19:54.909445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.909419    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-trusted-ca\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.909918 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.909872    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-installation-pull-secrets\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.910229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.910208    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-image-registry-private-configuration\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.910695 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.910676    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-serving-cert\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.914196 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.914175    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"]
Jul 27 23:19:54.920253 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.920228    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-kube-api-access-wzrjn\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:54.920427 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.920410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9f7j\" (UniqueName: \"kubernetes.io/projected/6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7-kube-api-access-s9f7j\") pod \"console-operator-584dcb6b54-tcw4n\" (UID: \"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7\") " pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:54.920607 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:54.920591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-bound-sa-token\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:55.008850 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.008749    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2z2\" (UniqueName: \"kubernetes.io/projected/a953c104-e172-4e38-8821-71efcc12d4e0-kube-api-access-xd2z2\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:55.008850 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.008810    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.009071 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.008877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:55.009071 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.008900    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qnw\" (UniqueName: \"kubernetes.io/projected/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-kube-api-access-78qnw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.009071 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.008931    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.009071 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.008967    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 23:19:55.009071 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.009025    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls podName:a953c104-e172-4e38-8821-71efcc12d4e0 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:55.509010098 +0000 UTC m=+220.460353463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls") pod "cluster-samples-operator-6748489656-5nqjg" (UID: "a953c104-e172-4e38-8821-71efcc12d4e0") : secret "samples-operator-tls" not found
Jul 27 23:19:55.019876 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.019843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2z2\" (UniqueName: \"kubernetes.io/projected/a953c104-e172-4e38-8821-71efcc12d4e0-kube-api-access-xd2z2\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:55.022670 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.022655    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:19:55.109862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.109831    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qnw\" (UniqueName: \"kubernetes.io/projected/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-kube-api-access-78qnw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.110020 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.109883    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.110020 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.109965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.110470 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.110452    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.112389 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.112367    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.118443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.118421    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qnw\" (UniqueName: \"kubernetes.io/projected/5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8-kube-api-access-78qnw\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7\" (UID: \"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.147152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.147116    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tcw4n"]
Jul 27 23:19:55.150627 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:19:55.150603    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7f7de0_e1cd_4854_90c2_b7c7c8ef28d7.slice/crio-eeeb9445a604ed17dd3035a824218126270395aff10819545fbfe57dcb2f4959 WatchSource:0}: Error finding container eeeb9445a604ed17dd3035a824218126270395aff10819545fbfe57dcb2f4959: Status 404 returned error can't find the container with id eeeb9445a604ed17dd3035a824218126270395aff10819545fbfe57dcb2f4959
Jul 27 23:19:55.199494 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.199455    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" event={"ID":"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7","Type":"ContainerStarted","Data":"eeeb9445a604ed17dd3035a824218126270395aff10819545fbfe57dcb2f4959"}
Jul 27 23:19:55.221339 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.221312    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"
Jul 27 23:19:55.344464 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.344433    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7"]
Jul 27 23:19:55.347159 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:19:55.347132    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8c2dee_8c8c_46e3_928d_fb8c42a78aa8.slice/crio-7e327d9f1fd3b635653b14a4d4f77c54d30c83690d403e9aaf8c870d2e1a67f7 WatchSource:0}: Error finding container 7e327d9f1fd3b635653b14a4d4f77c54d30c83690d403e9aaf8c870d2e1a67f7: Status 404 returned error can't find the container with id 7e327d9f1fd3b635653b14a4d4f77c54d30c83690d403e9aaf8c870d2e1a67f7
Jul 27 23:19:55.411708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.411675    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:55.411900 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.411847    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 23:19:55.411900 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.411872    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867c9589c8-fgvqc: secret "image-registry-tls" not found
Jul 27 23:19:55.412009 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.411943    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls podName:8cd51a6d-b85c-470a-9933-73fef8e40628 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:56.411920582 +0000 UTC m=+221.363263949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls") pod "image-registry-867c9589c8-fgvqc" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628") : secret "image-registry-tls" not found
Jul 27 23:19:55.512231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:55.512196    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:55.512398 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.512341    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 23:19:55.512463 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:55.512419    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls podName:a953c104-e172-4e38-8821-71efcc12d4e0 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:56.512403377 +0000 UTC m=+221.463746742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls") pod "cluster-samples-operator-6748489656-5nqjg" (UID: "a953c104-e172-4e38-8821-71efcc12d4e0") : secret "samples-operator-tls" not found
Jul 27 23:19:56.205264 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:56.205146    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7" event={"ID":"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8","Type":"ContainerStarted","Data":"7e327d9f1fd3b635653b14a4d4f77c54d30c83690d403e9aaf8c870d2e1a67f7"}
Jul 27 23:19:56.422878 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:56.422770    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:56.423059 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:56.422972    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 23:19:56.423059 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:56.422997    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867c9589c8-fgvqc: secret "image-registry-tls" not found
Jul 27 23:19:56.423059 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:56.423059    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls podName:8cd51a6d-b85c-470a-9933-73fef8e40628 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:58.423039073 +0000 UTC m=+223.374382443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls") pod "image-registry-867c9589c8-fgvqc" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628") : secret "image-registry-tls" not found
Jul 27 23:19:56.523778 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:56.523689    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:56.523935 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:56.523860    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 23:19:56.523935 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:56.523923    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls podName:a953c104-e172-4e38-8821-71efcc12d4e0 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:58.523904756 +0000 UTC m=+223.475248131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls") pod "cluster-samples-operator-6748489656-5nqjg" (UID: "a953c104-e172-4e38-8821-71efcc12d4e0") : secret "samples-operator-tls" not found
Jul 27 23:19:58.213232 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.213191    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7" event={"ID":"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8","Type":"ContainerStarted","Data":"306fdc077e9fd1168d04dfec63ab782c8eb44438ff824a96e884e3550a711373"}
Jul 27 23:19:58.214886 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.214868    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/0.log"
Jul 27 23:19:58.214988 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.214899    2570 generic.go:356] "Generic (PLEG): container finished" podID="6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7" containerID="6fce65de1f95ca8d97be0fc82eceebccb0ac0ea5f382e69ee3912bed4dd02904" exitCode=255
Jul 27 23:19:58.214988 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.214930    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" event={"ID":"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7","Type":"ContainerDied","Data":"6fce65de1f95ca8d97be0fc82eceebccb0ac0ea5f382e69ee3912bed4dd02904"}
Jul 27 23:19:58.215177 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.215160    2570 scope.go:117] "RemoveContainer" containerID="6fce65de1f95ca8d97be0fc82eceebccb0ac0ea5f382e69ee3912bed4dd02904"
Jul 27 23:19:58.230947 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.230895    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7" podStartSLOduration=1.9843904509999999 podStartE2EDuration="4.230873207s" totalImagesPullingTime="2.246482756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:19:58.22945043 +0000 UTC m=+223.180793834" watchObservedRunningTime="2026-07-27 23:19:58.230873207 +0000 UTC m=+223.182216603"
Jul 27 23:19:58.439739 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.439661    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:19:58.439932 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:58.439820    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 23:19:58.439932 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:58.439842    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867c9589c8-fgvqc: secret "image-registry-tls" not found
Jul 27 23:19:58.439932 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:58.439907    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls podName:8cd51a6d-b85c-470a-9933-73fef8e40628 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:02.439888734 +0000 UTC m=+227.391232099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls") pod "image-registry-867c9589c8-fgvqc" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628") : secret "image-registry-tls" not found
Jul 27 23:19:58.540194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:58.540106    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:19:58.540329 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:58.540276    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 23:19:58.540404 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:58.540377    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls podName:a953c104-e172-4e38-8821-71efcc12d4e0 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:02.540336615 +0000 UTC m=+227.491679987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls") pod "cluster-samples-operator-6748489656-5nqjg" (UID: "a953c104-e172-4e38-8821-71efcc12d4e0") : secret "samples-operator-tls" not found
Jul 27 23:19:59.219889 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.219856    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:19:59.220385 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.220346    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/0.log"
Jul 27 23:19:59.220462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.220414    2570 generic.go:356] "Generic (PLEG): container finished" podID="6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7" containerID="a3ec753c4082f35fc63ef73ceb689b3672f41f275f6b569bfa28f18af96558e7" exitCode=255
Jul 27 23:19:59.220546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.220520    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" event={"ID":"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7","Type":"ContainerDied","Data":"a3ec753c4082f35fc63ef73ceb689b3672f41f275f6b569bfa28f18af96558e7"}
Jul 27 23:19:59.220617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.220604    2570 scope.go:117] "RemoveContainer" containerID="6fce65de1f95ca8d97be0fc82eceebccb0ac0ea5f382e69ee3912bed4dd02904"
Jul 27 23:19:59.221020 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.220984    2570 scope.go:117] "RemoveContainer" containerID="a3ec753c4082f35fc63ef73ceb689b3672f41f275f6b569bfa28f18af96558e7"
Jul 27 23:19:59.221332 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:59.221147    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tcw4n_openshift-console-operator(6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" podUID="6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7"
Jul 27 23:19:59.222502 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.222482    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"]
Jul 27 23:19:59.225529 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.225498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"
Jul 27 23:19:59.228025 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.228006    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 23:19:59.228119 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.228038    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 23:19:59.228119 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.228110    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2nqxm\""
Jul 27 23:19:59.233981 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.233957    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"]
Jul 27 23:19:59.346154 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.346112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6v9\" (UniqueName: \"kubernetes.io/projected/53a03528-7f33-4d0e-9bad-2e2ffd268476-kube-api-access-sm6v9\") pod \"migrator-6865c9967-cjr89\" (UID: \"53a03528-7f33-4d0e-9bad-2e2ffd268476\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"
Jul 27 23:19:59.370309 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.370273    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-98hh2"]
Jul 27 23:19:59.374652 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.374629    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.379252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.379110    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 23:19:59.379252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.379126    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 23:19:59.379252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.379170    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 23:19:59.379252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.379183    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26z57\""
Jul 27 23:19:59.379252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.379197    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 23:19:59.386411 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.386388    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-98hh2"]
Jul 27 23:19:59.447497 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.447454    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6v9\" (UniqueName: \"kubernetes.io/projected/53a03528-7f33-4d0e-9bad-2e2ffd268476-kube-api-access-sm6v9\") pod \"migrator-6865c9967-cjr89\" (UID: \"53a03528-7f33-4d0e-9bad-2e2ffd268476\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"
Jul 27 23:19:59.447688 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.447510    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fnw6\" (UniqueName: \"kubernetes.io/projected/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-api-access-4fnw6\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.447688 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.447649    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.447688 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.447680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9790621d-3b93-4f60-aba5-b4a771373ea9-crio-socket\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.447846 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.447720    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.447846 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.447751    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9790621d-3b93-4f60-aba5-b4a771373ea9-data-volume\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.457856 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.457819    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6v9\" (UniqueName: \"kubernetes.io/projected/53a03528-7f33-4d0e-9bad-2e2ffd268476-kube-api-access-sm6v9\") pod \"migrator-6865c9967-cjr89\" (UID: \"53a03528-7f33-4d0e-9bad-2e2ffd268476\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"
Jul 27 23:19:59.535536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.535446    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"
Jul 27 23:19:59.548829 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.548803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.548937 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.548848    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9790621d-3b93-4f60-aba5-b4a771373ea9-crio-socket\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.548937 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.548920    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9790621d-3b93-4f60-aba5-b4a771373ea9-crio-socket\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.549048 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:59.548946    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:19:59.549048 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.548995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.549048 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:59.549031    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:00.049008416 +0000 UTC m=+225.000351780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:19:59.549183 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:59.549058    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:19:59.549183 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.549089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9790621d-3b93-4f60-aba5-b4a771373ea9-data-volume\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.549183 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:19:59.549123    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:00.04910673 +0000 UTC m=+225.000450095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : configmap "kube-rbac-proxy" not found
Jul 27 23:19:59.549339 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.549206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fnw6\" (UniqueName: \"kubernetes.io/projected/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-api-access-4fnw6\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.549420 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.549399    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9790621d-3b93-4f60-aba5-b4a771373ea9-data-volume\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.557197 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.557175    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fnw6\" (UniqueName: \"kubernetes.io/projected/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-api-access-4fnw6\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:19:59.662439 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:19:59.662409    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89"]
Jul 27 23:19:59.665445 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:19:59.665417    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a03528_7f33_4d0e_9bad_2e2ffd268476.slice/crio-e5047c3e3dd9bb903202c470f10577fedcbd96ece0f00cd626505f6952b7aae7 WatchSource:0}: Error finding container e5047c3e3dd9bb903202c470f10577fedcbd96ece0f00cd626505f6952b7aae7: Status 404 returned error can't find the container with id e5047c3e3dd9bb903202c470f10577fedcbd96ece0f00cd626505f6952b7aae7
Jul 27 23:20:00.052889 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:00.052848    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:00.053066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:00.052899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:00.053066 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:00.052998    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:00.053066 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:00.053010    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:00.053066 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:00.053052    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:01.053036153 +0000 UTC m=+226.004379519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:00.053312 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:00.053078    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:01.053062834 +0000 UTC m=+226.004406203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:00.225344 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:00.225313    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:20:00.225770 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:00.225746    2570 scope.go:117] "RemoveContainer" containerID="a3ec753c4082f35fc63ef73ceb689b3672f41f275f6b569bfa28f18af96558e7"
Jul 27 23:20:00.225952 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:00.225930    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tcw4n_openshift-console-operator(6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" podUID="6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7"
Jul 27 23:20:00.226535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:00.226512    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89" event={"ID":"53a03528-7f33-4d0e-9bad-2e2ffd268476","Type":"ContainerStarted","Data":"e5047c3e3dd9bb903202c470f10577fedcbd96ece0f00cd626505f6952b7aae7"}
Jul 27 23:20:01.063286 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:01.063254    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:01.063482 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:01.063303    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:01.063482 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:01.063427    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:01.063482 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:01.063429    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:01.063482 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:01.063484    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:03.063469315 +0000 UTC m=+228.014812680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:01.063693 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:01.063502    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:03.063492231 +0000 UTC m=+228.014835596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:01.231208 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:01.231169    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89" event={"ID":"53a03528-7f33-4d0e-9bad-2e2ffd268476","Type":"ContainerStarted","Data":"e9cab2f970588981bf91836fb768e0bdd4d21d14f625fa238af4b76e2f516bc5"}
Jul 27 23:20:01.231580 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:01.231235    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89" event={"ID":"53a03528-7f33-4d0e-9bad-2e2ffd268476","Type":"ContainerStarted","Data":"6253449e11200af88872b6fa90628e31e84ceea01b604458a45478b20ff9d435"}
Jul 27 23:20:01.247386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:01.247318    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-cjr89" podStartSLOduration=1.031807424 podStartE2EDuration="2.247304329s" totalImagesPullingTime="1.215496905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:01.24604297 +0000 UTC m=+226.197386369" watchObservedRunningTime="2026-07-27 23:20:01.247304329 +0000 UTC m=+226.198647716"
Jul 27 23:20:02.157101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:02.157069    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pc986_4a32c919-8faa-44fd-9e91-d696e209a766/dns-node-resolver/0.log"
Jul 27 23:20:02.476338 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:02.476238    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:02.476770 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:02.476430    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 23:20:02.476770 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:02.476448    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867c9589c8-fgvqc: secret "image-registry-tls" not found
Jul 27 23:20:02.476770 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:02.476505    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls podName:8cd51a6d-b85c-470a-9933-73fef8e40628 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:10.47648743 +0000 UTC m=+235.427830815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls") pod "image-registry-867c9589c8-fgvqc" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628") : secret "image-registry-tls" not found
Jul 27 23:20:02.577404 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:02.577344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:20:02.577593 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:02.577519    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 23:20:02.577665 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:02.577596    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls podName:a953c104-e172-4e38-8821-71efcc12d4e0 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:10.577575939 +0000 UTC m=+235.528919303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls") pod "cluster-samples-operator-6748489656-5nqjg" (UID: "a953c104-e172-4e38-8821-71efcc12d4e0") : secret "samples-operator-tls" not found
Jul 27 23:20:03.082183 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:03.082149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:03.082376 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:03.082198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:03.082376 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:03.082293    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:03.082376 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:03.082316    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:03.082376 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:03.082348    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:07.082333042 +0000 UTC m=+232.033676407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:03.082552 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:03.082399    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:07.082381419 +0000 UTC m=+232.033724787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:03.156704 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:03.156679    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nkb9d_b48ab19b-f661-4ae8-84fe-96685d29fcfb/node-ca/0.log"
Jul 27 23:20:03.757410 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:03.757382    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbc6r_d5fb248c-397e-4373-9e50-8c22438a3fb1/serve-healthcheck-canary/0.log"
Jul 27 23:20:04.357310 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:04.357269    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-cjr89_53a03528-7f33-4d0e-9bad-2e2ffd268476/migrator/0.log"
Jul 27 23:20:04.556379 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:04.556324    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-cjr89_53a03528-7f33-4d0e-9bad-2e2ffd268476/graceful-termination/0.log"
Jul 27 23:20:04.757644 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:04.757614    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7_5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8/kube-storage-version-migrator-operator/0.log"
Jul 27 23:20:05.023844 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:05.023765    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:20:05.023844 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:05.023821    2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:20:05.024199 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:05.024182    2570 scope.go:117] "RemoveContainer" containerID="a3ec753c4082f35fc63ef73ceb689b3672f41f275f6b569bfa28f18af96558e7"
Jul 27 23:20:05.024395 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:05.024348    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tcw4n_openshift-console-operator(6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" podUID="6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7"
Jul 27 23:20:07.116341 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:07.116305    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:07.116725 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:07.116376    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:07.116725 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:07.116468    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:07.116725 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:07.116512    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 23:20:07.116725 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:07.116525    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:15.116509604 +0000 UTC m=+240.067852970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : secret "insights-runtime-extractor-tls" not found
Jul 27 23:20:07.116725 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:07.116576    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm podName:9790621d-3b93-4f60-aba5-b4a771373ea9 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:15.116561985 +0000 UTC m=+240.067905353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-98hh2" (UID: "9790621d-3b93-4f60-aba5-b4a771373ea9") : configmap "kube-rbac-proxy" not found
Jul 27 23:20:10.541147 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.541102    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:10.543698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.543678    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"image-registry-867c9589c8-fgvqc\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") " pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:10.627845 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.627800    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:10.642087 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.642038    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:20:10.644858 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.644834    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a953c104-e172-4e38-8821-71efcc12d4e0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5nqjg\" (UID: \"a953c104-e172-4e38-8821-71efcc12d4e0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:20:10.717691 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.717656    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"
Jul 27 23:20:10.758809 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.758760    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-867c9589c8-fgvqc"]
Jul 27 23:20:10.763435 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:20:10.763401    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd51a6d_b85c_470a_9933_73fef8e40628.slice/crio-85ddef25027eb7c13939bb781ece783f38c37c2711e87c5f45ebc60b91bbb3c7 WatchSource:0}: Error finding container 85ddef25027eb7c13939bb781ece783f38c37c2711e87c5f45ebc60b91bbb3c7: Status 404 returned error can't find the container with id 85ddef25027eb7c13939bb781ece783f38c37c2711e87c5f45ebc60b91bbb3c7
Jul 27 23:20:10.848802 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:10.848773    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg"]
Jul 27 23:20:11.261544 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:11.261502    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg" event={"ID":"a953c104-e172-4e38-8821-71efcc12d4e0","Type":"ContainerStarted","Data":"2c6c61a43037c8f9cc1724783fa0d749a662f2ff514b9ab9e922fc50aba32f9b"}
Jul 27 23:20:11.262995 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:11.262971    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" event={"ID":"8cd51a6d-b85c-470a-9933-73fef8e40628","Type":"ContainerStarted","Data":"cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9"}
Jul 27 23:20:11.263124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:11.263010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" event={"ID":"8cd51a6d-b85c-470a-9933-73fef8e40628","Type":"ContainerStarted","Data":"85ddef25027eb7c13939bb781ece783f38c37c2711e87c5f45ebc60b91bbb3c7"}
Jul 27 23:20:11.263124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:11.263102    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:11.281857 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:11.281814    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" podStartSLOduration=17.281799471 podStartE2EDuration="17.281799471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:11.280620033 +0000 UTC m=+236.231963420" watchObservedRunningTime="2026-07-27 23:20:11.281799471 +0000 UTC m=+236.233142858"
Jul 27 23:20:13.271820 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:13.271788    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg" event={"ID":"a953c104-e172-4e38-8821-71efcc12d4e0","Type":"ContainerStarted","Data":"54f798ef2e30e02ba272f69062e411c291158d1785b30ab59efdd01f733152e4"}
Jul 27 23:20:13.272268 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:13.271845    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg" event={"ID":"a953c104-e172-4e38-8821-71efcc12d4e0","Type":"ContainerStarted","Data":"c47c0710b1c883b5bdcf7db6cb272fcbc051e808943b53062ff1b2c43bd0f874"}
Jul 27 23:20:13.286751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:13.286677    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5nqjg" podStartSLOduration=17.670750656 podStartE2EDuration="19.286663163s" totalImagesPullingTime="1.615912507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:13.285698638 +0000 UTC m=+238.237042025" watchObservedRunningTime="2026-07-27 23:20:13.286663163 +0000 UTC m=+238.238006614"
Jul 27 23:20:15.176148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:15.176100    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:15.176148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:15.176156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:15.176723 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:15.176642    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9790621d-3b93-4f60-aba5-b4a771373ea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:15.178609 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:15.178585    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9790621d-3b93-4f60-aba5-b4a771373ea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-98hh2\" (UID: \"9790621d-3b93-4f60-aba5-b4a771373ea9\") " pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:15.287964 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:15.287932    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-98hh2"
Jul 27 23:20:15.415534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:15.415506    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-98hh2"]
Jul 27 23:20:15.417926 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:20:15.417903    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9790621d_3b93_4f60_aba5_b4a771373ea9.slice/crio-0b294943fa7ac325239fa0c4ebab4bbd21188add9626c6f9c111dfcc9c0b5229 WatchSource:0}: Error finding container 0b294943fa7ac325239fa0c4ebab4bbd21188add9626c6f9c111dfcc9c0b5229: Status 404 returned error can't find the container with id 0b294943fa7ac325239fa0c4ebab4bbd21188add9626c6f9c111dfcc9c0b5229
Jul 27 23:20:16.284909 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:16.284747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98hh2" event={"ID":"9790621d-3b93-4f60-aba5-b4a771373ea9","Type":"ContainerStarted","Data":"0b294943fa7ac325239fa0c4ebab4bbd21188add9626c6f9c111dfcc9c0b5229"}
Jul 27 23:20:17.516865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:17.516837    2570 scope.go:117] "RemoveContainer" containerID="a3ec753c4082f35fc63ef73ceb689b3672f41f275f6b569bfa28f18af96558e7"
Jul 27 23:20:18.295220 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:18.295191    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:20:18.295498 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:18.295278    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" event={"ID":"6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7","Type":"ContainerStarted","Data":"c1d6e1de9b63ff3e5d28f399c9c3555d87fadbcee37b6515d9beb75217aecdc5"}
Jul 27 23:20:18.295687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:18.295644    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:20:18.301955 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:18.301928    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n"
Jul 27 23:20:18.311897 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:18.311845    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-tcw4n" podStartSLOduration=21.872016579 podStartE2EDuration="24.311826216s" totalImagesPullingTime="2.439809637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:18.311153309 +0000 UTC m=+243.262496738" watchObservedRunningTime="2026-07-27 23:20:18.311826216 +0000 UTC m=+243.263169605"
Jul 27 23:20:21.307486 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:21.307442    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98hh2" event={"ID":"9790621d-3b93-4f60-aba5-b4a771373ea9","Type":"ContainerStarted","Data":"f480ae02911573f318747816fc029db1a17703d82db28ac049885e4811fe6806"}
Jul 27 23:20:24.053747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.053718    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-867c9589c8-fgvqc"]
Jul 27 23:20:24.091527 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.091491    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8799d4584-9dth5"]
Jul 27 23:20:24.096775 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.096738    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.107497 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.107474    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8799d4584-9dth5"]
Jul 27 23:20:24.251862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251776    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-trusted-ca\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.251862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-bound-sa-token\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.251862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-installation-pull-secrets\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.252089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsnf4\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-kube-api-access-rsnf4\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.252089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251897    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-ca-trust-extracted\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.252089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-image-registry-private-configuration\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.252089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.251983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-registry-certificates\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.252089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.252004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-registry-tls\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.319222 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.319187    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98hh2" event={"ID":"9790621d-3b93-4f60-aba5-b4a771373ea9","Type":"ContainerStarted","Data":"c86623caab0df4838d2effd6cbe4a6962e41771c84be5d814f5b384121b458ef"}
Jul 27 23:20:24.353214 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353177    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-registry-certificates\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353214 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353221    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-registry-tls\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353433 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353383    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-trusted-ca\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-bound-sa-token\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353544 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353519    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-installation-pull-secrets\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353592 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353551    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsnf4\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-kube-api-access-rsnf4\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353592 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-ca-trust-extracted\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.353750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.353627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-image-registry-private-configuration\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.354134 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.354045    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-ca-trust-extracted\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.354134 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.354135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-registry-certificates\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.354329 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.354220    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-trusted-ca\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.355949 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.355913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-registry-tls\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.356068 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.355996    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-installation-pull-secrets\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.356124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.356068    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-image-registry-private-configuration\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.364150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.364122    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-bound-sa-token\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.364322 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.364300    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsnf4\" (UniqueName: \"kubernetes.io/projected/b2d41c9a-ceac-4f55-85ce-45bf2b8427e9-kube-api-access-rsnf4\") pod \"image-registry-8799d4584-9dth5\" (UID: \"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9\") " pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.405261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.405235    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:24.533181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:24.533099    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8799d4584-9dth5"]
Jul 27 23:20:24.535849 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:20:24.535821    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d41c9a_ceac_4f55_85ce_45bf2b8427e9.slice/crio-d7f9710118d08324ac22620b67a3e05d3ecc290de71fcc342a0fcc2a9ab50615 WatchSource:0}: Error finding container d7f9710118d08324ac22620b67a3e05d3ecc290de71fcc342a0fcc2a9ab50615: Status 404 returned error can't find the container with id d7f9710118d08324ac22620b67a3e05d3ecc290de71fcc342a0fcc2a9ab50615
Jul 27 23:20:25.324678 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:25.324639    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8799d4584-9dth5" event={"ID":"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9","Type":"ContainerStarted","Data":"0595eaf3472010385fdd965c278e1acaf31631de430fddc9ce34c477077b77d5"}
Jul 27 23:20:25.325161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:25.324736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8799d4584-9dth5" event={"ID":"b2d41c9a-ceac-4f55-85ce-45bf2b8427e9","Type":"ContainerStarted","Data":"d7f9710118d08324ac22620b67a3e05d3ecc290de71fcc342a0fcc2a9ab50615"}
Jul 27 23:20:25.325161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:25.324814    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:25.343099 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:25.343044    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8799d4584-9dth5" podStartSLOduration=1.343025526 podStartE2EDuration="1.343025526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:25.34230378 +0000 UTC m=+250.293647169" watchObservedRunningTime="2026-07-27 23:20:25.343025526 +0000 UTC m=+250.294368914"
Jul 27 23:20:27.378945 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:27.378911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:20:27.381682 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:27.381650    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3acd34a-68bc-472f-a6e1-8169dbed08db-metrics-certs\") pod \"network-metrics-daemon-vbgm4\" (UID: \"b3acd34a-68bc-472f-a6e1-8169dbed08db\") " pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:20:27.519850 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:27.519814    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n98qx\""
Jul 27 23:20:27.528119 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:27.528094    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vbgm4"
Jul 27 23:20:27.717986 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:27.717959    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vbgm4"]
Jul 27 23:20:27.720694 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:20:27.720664    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3acd34a_68bc_472f_a6e1_8169dbed08db.slice/crio-35563e1d87b29ff5e1c893819a191b57933a8c36ed17ca9d04a00add8136c8a0 WatchSource:0}: Error finding container 35563e1d87b29ff5e1c893819a191b57933a8c36ed17ca9d04a00add8136c8a0: Status 404 returned error can't find the container with id 35563e1d87b29ff5e1c893819a191b57933a8c36ed17ca9d04a00add8136c8a0
Jul 27 23:20:28.338809 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:28.338765    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-98hh2" event={"ID":"9790621d-3b93-4f60-aba5-b4a771373ea9","Type":"ContainerStarted","Data":"c9280f0aa7a5cb986007aa57141e7cffdcb3aa6b779d677abd90326b31d26ae4"}
Jul 27 23:20:28.340268 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:28.340233    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbgm4" event={"ID":"b3acd34a-68bc-472f-a6e1-8169dbed08db","Type":"ContainerStarted","Data":"35563e1d87b29ff5e1c893819a191b57933a8c36ed17ca9d04a00add8136c8a0"}
Jul 27 23:20:28.357284 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:28.357236    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-98hh2" podStartSLOduration=17.257391141 podStartE2EDuration="29.357211654s" totalImagesPullingTime="12.099820513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:19:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:28.357107744 +0000 UTC m=+253.308451131" watchObservedRunningTime="2026-07-27 23:20:28.357211654 +0000 UTC m=+253.308555040"
Jul 27 23:20:29.345298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:29.345257    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbgm4" event={"ID":"b3acd34a-68bc-472f-a6e1-8169dbed08db","Type":"ContainerStarted","Data":"66c08620ba34071576f90622e88c36e2423c0dd17f1b37847b50ef92abc44b63"}
Jul 27 23:20:29.345721 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:29.345323    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vbgm4" event={"ID":"b3acd34a-68bc-472f-a6e1-8169dbed08db","Type":"ContainerStarted","Data":"d388debbc1d89874d6eee67652c2f56fbbfcd457fb26147d4505c789a55609fd"}
Jul 27 23:20:29.361317 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:29.361266    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vbgm4" podStartSLOduration=252.386287218 podStartE2EDuration="4m13.361247539s" totalImagesPullingTime="974.960321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:29.360503494 +0000 UTC m=+254.311846883" watchObservedRunningTime="2026-07-27 23:20:29.361247539 +0000 UTC m=+254.312590926"
Jul 27 23:20:32.757526 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.757490    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7sqsm"]
Jul 27 23:20:32.761149 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.761129    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.764468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764444    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"]
Jul 27 23:20:32.764810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764736    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2gwgm\""
Jul 27 23:20:32.764810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764753    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 23:20:32.764810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764756    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 23:20:32.764810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764763    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 23:20:32.765047 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764753    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 23:20:32.765047 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764872    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 23:20:32.765047 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.764994    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 23:20:32.767216 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.767198    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.771096 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.771078    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 23:20:32.771193 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.771149    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fn5vm\""
Jul 27 23:20:32.771261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.771243    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 23:20:32.771529 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.771512    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 23:20:32.777091 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.777075    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"]
Jul 27 23:20:32.821380 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821316    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-tls\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.821564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc9dd18c-2d32-4756-b686-d3cf65fcce48-metrics-client-ca\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-root\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821546    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsd8\" (UniqueName: \"kubernetes.io/projected/dc9dd18c-2d32-4756-b686-d3cf65fcce48-kube-api-access-slsd8\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-wtmp\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821598    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.821755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821619    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjkp8\" (UniqueName: \"kubernetes.io/projected/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-api-access-kjkp8\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.821755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.821755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821688    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.821755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.821939 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821762    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-sys\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821939 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821780    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.821939 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.821798    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-textfile\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922422 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922378    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.922583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922440    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc9dd18c-2d32-4756-b686-d3cf65fcce48-metrics-client-ca\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922471    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-root\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slsd8\" (UniqueName: \"kubernetes.io/projected/dc9dd18c-2d32-4756-b686-d3cf65fcce48-kube-api-access-slsd8\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922531    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-wtmp\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922577    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-root\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjkp8\" (UniqueName: \"kubernetes.io/projected/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-api-access-kjkp8\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922620    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922656    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922730    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-sys\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922734    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-wtmp\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922781    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922788    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc9dd18c-2d32-4756-b686-d3cf65fcce48-sys\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-textfile\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-tls\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.922935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.922914    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.923154    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc9dd18c-2d32-4756-b686-d3cf65fcce48-metrics-client-ca\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.923160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.923257    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:32.923260    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.923296    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:32.923343    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-tls podName:dc9dd18c-2d32-4756-b686-d3cf65fcce48 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:33.423323607 +0000 UTC m=+258.374666972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-tls") pod "node-exporter-7sqsm" (UID: "dc9dd18c-2d32-4756-b686-d3cf65fcce48") : secret "node-exporter-tls" not found
Jul 27 23:20:32.923542 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.923430    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-textfile\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.923776 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.923742    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-accelerators-collector-config\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.925386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.925341    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.925974 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.925952    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.926022 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.925952    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:32.930473 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.930455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsd8\" (UniqueName: \"kubernetes.io/projected/dc9dd18c-2d32-4756-b686-d3cf65fcce48-kube-api-access-slsd8\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:32.931092 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:32.931073    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjkp8\" (UniqueName: \"kubernetes.io/projected/d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd-kube-api-access-kjkp8\") pod \"kube-state-metrics-56d9f944b4-954qf\" (UID: \"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:33.078836 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.078731    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"
Jul 27 23:20:33.219114 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.215807    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-954qf"]
Jul 27 23:20:33.223041 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:20:33.223014    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a4ff3f_82f4_42f0_a54d_e447bcc8edbd.slice/crio-83fa1708eb0d7bf3d31843241281b3f9bd5fa41a8d8ff8189fd01ab77a5a7bfc WatchSource:0}: Error finding container 83fa1708eb0d7bf3d31843241281b3f9bd5fa41a8d8ff8189fd01ab77a5a7bfc: Status 404 returned error can't find the container with id 83fa1708eb0d7bf3d31843241281b3f9bd5fa41a8d8ff8189fd01ab77a5a7bfc
Jul 27 23:20:33.361398 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.361293    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf" event={"ID":"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd","Type":"ContainerStarted","Data":"83fa1708eb0d7bf3d31843241281b3f9bd5fa41a8d8ff8189fd01ab77a5a7bfc"}
Jul 27 23:20:33.427389 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.427328    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-tls\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:33.429849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.429823    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dc9dd18c-2d32-4756-b686-d3cf65fcce48-node-exporter-tls\") pod \"node-exporter-7sqsm\" (UID: \"dc9dd18c-2d32-4756-b686-d3cf65fcce48\") " pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:33.671663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.671624    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7sqsm"
Jul 27 23:20:33.683795 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:20:33.683764    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9dd18c_2d32_4756_b686_d3cf65fcce48.slice/crio-182d92af7e8078a1c9adcede2e29f5249a8949af7fdf0668c81ebd81427ad2a0 WatchSource:0}: Error finding container 182d92af7e8078a1c9adcede2e29f5249a8949af7fdf0668c81ebd81427ad2a0: Status 404 returned error can't find the container with id 182d92af7e8078a1c9adcede2e29f5249a8949af7fdf0668c81ebd81427ad2a0
Jul 27 23:20:33.800714 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.800681    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:20:33.804798 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.804779    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.807496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.807469    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 23:20:33.807627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.807517    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 23:20:33.807696 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.807649    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 23:20:33.807761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.807744    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 23:20:33.807808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.807772    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 23:20:33.807849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.807842    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 23:20:33.808020 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.808001    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 23:20:33.808112 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.808021    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g6c2p\""
Jul 27 23:20:33.808112 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.808033    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 23:20:33.808112 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.808043    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 23:20:33.816178 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.816158    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:20:33.832116 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832086    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832260 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832124    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832260 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832155    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832260 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832317    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832335    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-web-config\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832428    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832676 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z45q\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-kube-api-access-7z45q\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832676 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832518    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-config-out\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832676 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832548    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.832676 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.832577    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-config-volume\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.933750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933671    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.933750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.933750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933752    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-web-config\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933772    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933812    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z45q\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-kube-api-access-7z45q\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-config-out\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933919    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-config-volume\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.933974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.934009    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.934042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.934510 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.934077    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.935468 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:33.934994    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle podName:0f2749ac-c411-4947-92b6-1050c4bb1746 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:34.434967962 +0000 UTC m=+259.386311344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746") : configmap references non-existent config key: ca-bundle.crt
Jul 27 23:20:33.935468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.935157    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.936420 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.935870    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.938021 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.937963    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-config-out\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.938230 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.938115    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.939273 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.939246    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.939591 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.939568    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-web-config\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.939875 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.939853    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.940009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.939986    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.940525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.940500    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.940899 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.940876    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.940998 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.940944    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-config-volume\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:33.943266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:33.943243    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z45q\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-kube-api-access-7z45q\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:34.060536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:34.060497    2570 patch_prober.go:28] interesting pod/image-registry-867c9589c8-fgvqc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 23:20:34.060839 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:34.060563    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:20:34.368031 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:34.367940    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sqsm" event={"ID":"dc9dd18c-2d32-4756-b686-d3cf65fcce48","Type":"ContainerStarted","Data":"182d92af7e8078a1c9adcede2e29f5249a8949af7fdf0668c81ebd81427ad2a0"}
Jul 27 23:20:34.437663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:34.437629    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:34.438419 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:34.438396    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:34.714789 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:34.714743    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:20:35.010673 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.008937    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:20:35.373799 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.373709    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf" event={"ID":"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd","Type":"ContainerStarted","Data":"f53705b3334ce0dd3b1fa3a2ae37d18a2c426b387d5f6df415ad3017d408342d"}
Jul 27 23:20:35.373799 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.373770    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf" event={"ID":"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd","Type":"ContainerStarted","Data":"87afbbe9402ea88871ad7601ee00a8f9a7a7b8137402e6ec83e4a88890bb27ad"}
Jul 27 23:20:35.374027 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.373802    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf" event={"ID":"d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd","Type":"ContainerStarted","Data":"5413e33befb0ffdd7cb18b2510f9e33ef17ca5433d0421d4415b138298512ee9"}
Jul 27 23:20:35.374920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.374892    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"3742c8ead44c4f400aac2b6c1fde8fad397da9a0faf65f79d80d700b6b11724b"}
Jul 27 23:20:35.376321 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.376298    2570 generic.go:356] "Generic (PLEG): container finished" podID="dc9dd18c-2d32-4756-b686-d3cf65fcce48" containerID="42d9fe0b1145a75bf00770bb058752090b30f81eaa46e5380505e3b82c67c8a2" exitCode=0
Jul 27 23:20:35.376438 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.376378    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sqsm" event={"ID":"dc9dd18c-2d32-4756-b686-d3cf65fcce48","Type":"ContainerDied","Data":"42d9fe0b1145a75bf00770bb058752090b30f81eaa46e5380505e3b82c67c8a2"}
Jul 27 23:20:35.414455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:35.414401    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-954qf" podStartSLOduration=1.7844645350000001 podStartE2EDuration="3.414379046s" totalImagesPullingTime="1.629914511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:35.412260496 +0000 UTC m=+260.363603908" watchObservedRunningTime="2026-07-27 23:20:35.414379046 +0000 UTC m=+260.365722434"
Jul 27 23:20:36.381681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:36.381577    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a" exitCode=0
Jul 27 23:20:36.381681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:36.381662    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"}
Jul 27 23:20:36.383932 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:36.383895    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sqsm" event={"ID":"dc9dd18c-2d32-4756-b686-d3cf65fcce48","Type":"ContainerStarted","Data":"5d0abffb5437cab4fd8841ae41bc00c7377e3640d777fb577b5529960109cc40"}
Jul 27 23:20:36.384062 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:36.383954    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7sqsm" event={"ID":"dc9dd18c-2d32-4756-b686-d3cf65fcce48","Type":"ContainerStarted","Data":"ce17f8ebca9727bb551852212dc23009a00e6d6d29eefade0f9ae1231740e235"}
Jul 27 23:20:36.420243 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:36.420186    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7sqsm" podStartSLOduration=3.134525142 podStartE2EDuration="4.420170158s" totalImagesPullingTime="1.171391758s" totalInitContainerRuntime="114.253258ms" isStatefulPod=true podCreationTimestamp="2026-07-27 23:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:36.418565817 +0000 UTC m=+261.369909224" watchObservedRunningTime="2026-07-27 23:20:36.420170158 +0000 UTC m=+261.371513548"
Jul 27 23:20:38.394101 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:38.394069    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"}
Jul 27 23:20:38.394471 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:38.394129    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"}
Jul 27 23:20:38.394471 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:38.394155    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"}
Jul 27 23:20:38.394471 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:38.394179    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"}
Jul 27 23:20:38.394471 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:38.394202    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"}
Jul 27 23:20:39.401759 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:39.401661    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerStarted","Data":"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"}
Jul 27 23:20:39.428005 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:39.427955    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.845929558 podStartE2EDuration="6.427938071s" totalImagesPullingTime="3.566859224s" totalInitContainerRuntime="15.149289ms" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:39.427174703 +0000 UTC m=+264.378518103" watchObservedRunningTime="2026-07-27 23:20:39.427938071 +0000 UTC m=+264.379281457"
Jul 27 23:20:44.058978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:44.058945    2570 patch_prober.go:28] interesting pod/image-registry-867c9589c8-fgvqc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 23:20:44.059406 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:44.058995    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:20:44.410038 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:44.410009    2570 patch_prober.go:28] interesting pod/image-registry-8799d4584-9dth5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 23:20:44.410203 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:44.410063    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8799d4584-9dth5" podUID="b2d41c9a-ceac-4f55-85ce-45bf2b8427e9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:20:46.332788 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:46.332750    2570 patch_prober.go:28] interesting pod/image-registry-8799d4584-9dth5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 23:20:46.333140 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:46.332802    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8799d4584-9dth5" podUID="b2d41c9a-ceac-4f55-85ce-45bf2b8427e9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:20:49.072662 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.072612    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerName="registry" containerID="cri-o://cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9" gracePeriod=30
Jul 27 23:20:49.365195 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.365157    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:49.438993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.438963    2570 generic.go:356] "Generic (PLEG): container finished" podID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerID="cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9" exitCode=0
Jul 27 23:20:49.439163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.439016    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc"
Jul 27 23:20:49.439163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.439057    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" event={"ID":"8cd51a6d-b85c-470a-9933-73fef8e40628","Type":"ContainerDied","Data":"cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9"}
Jul 27 23:20:49.439163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.439111    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867c9589c8-fgvqc" event={"ID":"8cd51a6d-b85c-470a-9933-73fef8e40628","Type":"ContainerDied","Data":"85ddef25027eb7c13939bb781ece783f38c37c2711e87c5f45ebc60b91bbb3c7"}
Jul 27 23:20:49.439163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.439139    2570 scope.go:117] "RemoveContainer" containerID="cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9"
Jul 27 23:20:49.447455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.447429    2570 scope.go:117] "RemoveContainer" containerID="cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9"
Jul 27 23:20:49.447782 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:49.447762    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9\": container with ID starting with cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9 not found: ID does not exist" containerID="cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9"
Jul 27 23:20:49.447834 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.447793    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9"} err="failed to get container status \"cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9\": rpc error: code = NotFound desc = could not find container \"cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9\": container with ID starting with cd650b015c816de92312d70189c2a558288b574e73826bfc9d42b8ffdedf92d9 not found: ID does not exist"
Jul 27 23:20:49.467096 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467062    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd51a6d-b85c-470a-9933-73fef8e40628-ca-trust-extracted\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467106    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-installation-pull-secrets\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467140    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-certificates\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467175    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-kube-api-access-wzrjn\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467200    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467234    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-trusted-ca\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467325    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-image-registry-private-configuration\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467420    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-bound-sa-token\") pod \"8cd51a6d-b85c-470a-9933-73fef8e40628\" (UID: \"8cd51a6d-b85c-470a-9933-73fef8e40628\") "
Jul 27 23:20:49.467674 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467648    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:20:49.467732 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.467696    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:20:49.469790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.469753    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:20:49.469904 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.469795    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:20:49.470015 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.469990    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-kube-api-access-wzrjn" (OuterVolumeSpecName: "kube-api-access-wzrjn") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "kube-api-access-wzrjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:20:49.470104 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.470082    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:20:49.470208 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.470189    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:20:49.476055 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.476029    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd51a6d-b85c-470a-9933-73fef8e40628-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8cd51a6d-b85c-470a-9933-73fef8e40628" (UID: "8cd51a6d-b85c-470a-9933-73fef8e40628"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:20:49.568752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568714    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-certificates\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.568752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568743    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrjn\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-kube-api-access-wzrjn\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.568752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568753    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-registry-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.568752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568761    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cd51a6d-b85c-470a-9933-73fef8e40628-trusted-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.569061 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568770    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-image-registry-private-configuration\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.569061 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568780    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cd51a6d-b85c-470a-9933-73fef8e40628-bound-sa-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.569061 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568788    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cd51a6d-b85c-470a-9933-73fef8e40628-ca-trust-extracted\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.569061 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.568796    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cd51a6d-b85c-470a-9933-73fef8e40628-installation-pull-secrets\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:20:49.758229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.755546    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-867c9589c8-fgvqc"]
Jul 27 23:20:49.760857 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:49.760825    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-867c9589c8-fgvqc"]
Jul 27 23:20:51.520626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:51.520595    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" path="/var/lib/kubelet/pods/8cd51a6d-b85c-470a-9933-73fef8e40628/volumes"
Jul 27 23:20:54.410020 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:54.409988    2570 patch_prober.go:28] interesting pod/image-registry-8799d4584-9dth5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 23:20:54.410399 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:54.410071    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8799d4584-9dth5" podUID="b2d41c9a-ceac-4f55-85ce-45bf2b8427e9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:20:56.333373 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:56.333330    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8799d4584-9dth5"
Jul 27 23:20:58.051347 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:20:58.051302    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zw7cg" podUID="fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c"
Jul 27 23:20:58.474866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:20:58.474838    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:21:01.367697 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:01.367656    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:21:01.370219 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:01.370194    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c-metrics-tls\") pod \"dns-default-zw7cg\" (UID: \"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c\") " pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:21:01.478447 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:01.478418    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qdw\""
Jul 27 23:21:01.485978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:01.485961    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:21:01.650506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:01.650481    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zw7cg"]
Jul 27 23:21:01.652912 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:21:01.652880    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5f0f84_3c54_4f0b_a9e3_58318ffdb74c.slice/crio-e341f7a96268922dfe1aaba63400a677c7885e3fc5a0fe4ff6ddb9d836584957 WatchSource:0}: Error finding container e341f7a96268922dfe1aaba63400a677c7885e3fc5a0fe4ff6ddb9d836584957: Status 404 returned error can't find the container with id e341f7a96268922dfe1aaba63400a677c7885e3fc5a0fe4ff6ddb9d836584957
Jul 27 23:21:02.496130 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:02.496086    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw7cg" event={"ID":"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c","Type":"ContainerStarted","Data":"e341f7a96268922dfe1aaba63400a677c7885e3fc5a0fe4ff6ddb9d836584957"}
Jul 27 23:21:03.505620 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:03.505581    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw7cg" event={"ID":"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c","Type":"ContainerStarted","Data":"2b555101e2bcc6e36b07e1989e389a03033ae26d2263aa23f0f61c70a33823f8"}
Jul 27 23:21:04.510567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:04.510527    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zw7cg" event={"ID":"fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c","Type":"ContainerStarted","Data":"ff646f96c83595a62e61bd9fe8395f40c8e0f43b65dea9fb90b3ee6e36093849"}
Jul 27 23:21:04.511026 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:04.510625    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:21:04.527448 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:04.527397    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zw7cg" podStartSLOduration=251.777129567 podStartE2EDuration="4m13.527380617s" totalImagesPullingTime="1.75025105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:21:04.525622514 +0000 UTC m=+289.476965929" watchObservedRunningTime="2026-07-27 23:21:04.527380617 +0000 UTC m=+289.478724001"
Jul 27 23:21:14.516226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:14.516197    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zw7cg"
Jul 27 23:21:15.362745 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:15.362713    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:21:15.366864 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:15.366844    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:21:15.374301 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:15.374280    2570 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 23:21:29.609183 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:29.609148    2570 generic.go:356] "Generic (PLEG): container finished" podID="5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8" containerID="306fdc077e9fd1168d04dfec63ab782c8eb44438ff824a96e884e3550a711373" exitCode=0
Jul 27 23:21:29.609605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:29.609222    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7" event={"ID":"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8","Type":"ContainerDied","Data":"306fdc077e9fd1168d04dfec63ab782c8eb44438ff824a96e884e3550a711373"}
Jul 27 23:21:29.609605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:29.609598    2570 scope.go:117] "RemoveContainer" containerID="306fdc077e9fd1168d04dfec63ab782c8eb44438ff824a96e884e3550a711373"
Jul 27 23:21:29.862226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:29.862132    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbc6r_d5fb248c-397e-4373-9e50-8c22438a3fb1/serve-healthcheck-canary/0.log"
Jul 27 23:21:30.618120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:30.618079    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7" event={"ID":"5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8","Type":"ContainerStarted","Data":"40b423ccf8384ee0ac6246125d6e24203791548bf5d991f5e52176eb876c4583"}
Jul 27 23:21:52.980801 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.980762    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:21:52.981426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.981158    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="alertmanager" containerID="cri-o://ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd" gracePeriod=120
Jul 27 23:21:52.981426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.981221    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-metric" containerID="cri-o://b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec" gracePeriod=120
Jul 27 23:21:52.981426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.981228    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-web" containerID="cri-o://dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798" gracePeriod=120
Jul 27 23:21:52.981426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.981259    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy" containerID="cri-o://1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58" gracePeriod=120
Jul 27 23:21:52.981426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.981294    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="config-reloader" containerID="cri-o://3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67" gracePeriod=120
Jul 27 23:21:52.981685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:52.981613    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="prom-label-proxy" containerID="cri-o://2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994" gracePeriod=120
Jul 27 23:21:53.708648 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708615    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994" exitCode=0
Jul 27 23:21:53.708648 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708640    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58" exitCode=0
Jul 27 23:21:53.708648 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708647    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67" exitCode=0
Jul 27 23:21:53.708648 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708655    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd" exitCode=0
Jul 27 23:21:53.708905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708691    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"}
Jul 27 23:21:53.708905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708754    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"}
Jul 27 23:21:53.708905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708780    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"}
Jul 27 23:21:53.708905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:53.708804    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"}
Jul 27 23:21:54.220416 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.220383    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.281176 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281146    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281176 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281181    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z45q\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-kube-api-access-7z45q\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281203    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-metrics-client-ca\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281250    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-main-tls\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281313    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-tls-assets\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281347    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-web\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281416    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-config-volume\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281459    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-cluster-tls-config\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281491    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-web-config\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281513    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-config-out\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281549    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281584    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.281920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.281618    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-main-db\") pod \"0f2749ac-c411-4947-92b6-1050c4bb1746\" (UID: \"0f2749ac-c411-4947-92b6-1050c4bb1746\") "
Jul 27 23:21:54.282308 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.282276    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:21:54.283298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.283098    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:54.283298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.283260    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:21:54.285455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.285411    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.285743 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.285701    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:21:54.286205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.286177    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.286300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.286203    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.286300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.286290    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.286776 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.286727    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-config-volume" (OuterVolumeSpecName: "config-volume") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.286776 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.286741    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-kube-api-access-7z45q" (OuterVolumeSpecName: "kube-api-access-7z45q") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "kube-api-access-7z45q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:21:54.287473 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.287439    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-config-out" (OuterVolumeSpecName: "config-out") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:21:54.289916 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.289759    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.298262 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.298241    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-web-config" (OuterVolumeSpecName: "web-config") pod "0f2749ac-c411-4947-92b6-1050c4bb1746" (UID: "0f2749ac-c411-4947-92b6-1050c4bb1746"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:21:54.382731 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382677    2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-tls-assets\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.382731 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382722    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.382731 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382733    2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.382731 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382744    2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-cluster-tls-config\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.382731 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382752    2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-web-config\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382760    2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-config-out\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382769    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382779    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382787    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f2749ac-c411-4947-92b6-1050c4bb1746-alertmanager-main-db\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382796    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382804    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z45q\" (UniqueName: \"kubernetes.io/projected/0f2749ac-c411-4947-92b6-1050c4bb1746-kube-api-access-7z45q\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382818    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f2749ac-c411-4947-92b6-1050c4bb1746-metrics-client-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.383035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.382826    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f2749ac-c411-4947-92b6-1050c4bb1746-secret-alertmanager-main-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:21:54.716458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716425    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec" exitCode=0
Jul 27 23:21:54.716458 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716453    2570 generic.go:356] "Generic (PLEG): container finished" podID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerID="dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798" exitCode=0
Jul 27 23:21:54.716655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716501    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.716655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716514    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"}
Jul 27 23:21:54.716655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716576    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"}
Jul 27 23:21:54.716655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716604    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f2749ac-c411-4947-92b6-1050c4bb1746","Type":"ContainerDied","Data":"3742c8ead44c4f400aac2b6c1fde8fad397da9a0faf65f79d80d700b6b11724b"}
Jul 27 23:21:54.716655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.716637    2570 scope.go:117] "RemoveContainer" containerID="2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"
Jul 27 23:21:54.725171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.725155    2570 scope.go:117] "RemoveContainer" containerID="b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"
Jul 27 23:21:54.732327 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.732305    2570 scope.go:117] "RemoveContainer" containerID="1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"
Jul 27 23:21:54.738461 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.738438    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:21:54.739430 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.739416    2570 scope.go:117] "RemoveContainer" containerID="dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"
Jul 27 23:21:54.743010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.742990    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:21:54.746883 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.746859    2570 scope.go:117] "RemoveContainer" containerID="3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"
Jul 27 23:21:54.753638 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.753618    2570 scope.go:117] "RemoveContainer" containerID="ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"
Jul 27 23:21:54.760817 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.760800    2570 scope.go:117] "RemoveContainer" containerID="f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"
Jul 27 23:21:54.767693 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.767664    2570 scope.go:117] "RemoveContainer" containerID="2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"
Jul 27 23:21:54.768011 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.767985    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994\": container with ID starting with 2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994 not found: ID does not exist" containerID="2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"
Jul 27 23:21:54.768114 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.768023    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"} err="failed to get container status \"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994\": rpc error: code = NotFound desc = could not find container \"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994\": container with ID starting with 2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994 not found: ID does not exist"
Jul 27 23:21:54.768114 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.768050    2570 scope.go:117] "RemoveContainer" containerID="b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"
Jul 27 23:21:54.768437 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.768408    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec\": container with ID starting with b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec not found: ID does not exist" containerID="b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"
Jul 27 23:21:54.768537 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.768446    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"} err="failed to get container status \"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec\": rpc error: code = NotFound desc = could not find container \"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec\": container with ID starting with b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec not found: ID does not exist"
Jul 27 23:21:54.768537 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.768468    2570 scope.go:117] "RemoveContainer" containerID="1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"
Jul 27 23:21:54.768772 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.768753    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58\": container with ID starting with 1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58 not found: ID does not exist" containerID="1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"
Jul 27 23:21:54.768818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.768777    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"} err="failed to get container status \"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58\": rpc error: code = NotFound desc = could not find container \"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58\": container with ID starting with 1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58 not found: ID does not exist"
Jul 27 23:21:54.768818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.768793    2570 scope.go:117] "RemoveContainer" containerID="dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"
Jul 27 23:21:54.769008 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.768990    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798\": container with ID starting with dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798 not found: ID does not exist" containerID="dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"
Jul 27 23:21:54.769075 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769016    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"} err="failed to get container status \"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798\": rpc error: code = NotFound desc = could not find container \"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798\": container with ID starting with dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798 not found: ID does not exist"
Jul 27 23:21:54.769075 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769035    2570 scope.go:117] "RemoveContainer" containerID="3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"
Jul 27 23:21:54.769265 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.769248    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67\": container with ID starting with 3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67 not found: ID does not exist" containerID="3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"
Jul 27 23:21:54.769331 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769272    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"} err="failed to get container status \"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67\": rpc error: code = NotFound desc = could not find container \"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67\": container with ID starting with 3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67 not found: ID does not exist"
Jul 27 23:21:54.769331 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769290    2570 scope.go:117] "RemoveContainer" containerID="ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"
Jul 27 23:21:54.769442 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769396    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:21:54.769539 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.769518    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd\": container with ID starting with ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd not found: ID does not exist" containerID="ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"
Jul 27 23:21:54.769581 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769544    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"} err="failed to get container status \"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd\": rpc error: code = NotFound desc = could not find container \"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd\": container with ID starting with ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd not found: ID does not exist"
Jul 27 23:21:54.769581 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769556    2570 scope.go:117] "RemoveContainer" containerID="f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"
Jul 27 23:21:54.769649 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769640    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="init-config-reloader"
Jul 27 23:21:54.769684 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769655    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="init-config-reloader"
Jul 27 23:21:54.769684 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769665    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="alertmanager"
Jul 27 23:21:54.769684 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769674    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="alertmanager"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769685    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-metric"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769691    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-metric"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769702    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerName="registry"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769707    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerName="registry"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769718    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769725    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769735    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="config-reloader"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769740    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="config-reloader"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769749    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-web"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769757    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-web"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769764    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="prom-label-proxy"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769769    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="prom-label-proxy"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:21:54.769796    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a\": container with ID starting with f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a not found: ID does not exist" containerID="f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769811    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="alertmanager"
Jul 27 23:21:54.769812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769818    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="prom-label-proxy"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769825    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-web"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769821    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"} err="failed to get container status \"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a\": rpc error: code = NotFound desc = could not find container \"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a\": container with ID starting with f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a not found: ID does not exist"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769847    2570 scope.go:117] "RemoveContainer" containerID="2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769831    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="config-reloader"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769908    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy-metric"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769920    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd51a6d-b85c-470a-9933-73fef8e40628" containerName="registry"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.769930    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" containerName="kube-rbac-proxy"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770079    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994"} err="failed to get container status \"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994\": rpc error: code = NotFound desc = could not find container \"2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994\": container with ID starting with 2515bfcc1c288286471eecf7b065b174df6b5062d8c2aac39d1152bd365c3994 not found: ID does not exist"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770094    2570 scope.go:117] "RemoveContainer" containerID="b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770320    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec"} err="failed to get container status \"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec\": rpc error: code = NotFound desc = could not find container \"b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec\": container with ID starting with b6f65865c16c198c7928c3c30c00df3e93d336f2f6126d3c689bbd9c0c929aec not found: ID does not exist"
Jul 27 23:21:54.770534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770338    2570 scope.go:117] "RemoveContainer" containerID="1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"
Jul 27 23:21:54.771040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770570    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58"} err="failed to get container status \"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58\": rpc error: code = NotFound desc = could not find container \"1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58\": container with ID starting with 1b0097c4750075e0f08c256028e843982a1de1410be6b8be11612e79950b4d58 not found: ID does not exist"
Jul 27 23:21:54.771040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770587    2570 scope.go:117] "RemoveContainer" containerID="dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"
Jul 27 23:21:54.771040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770774    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798"} err="failed to get container status \"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798\": rpc error: code = NotFound desc = could not find container \"dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798\": container with ID starting with dc35fdd39d50fcbee984cc5c3e775792bb691e999735735c3074dc4f295a6798 not found: ID does not exist"
Jul 27 23:21:54.771040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.770792    2570 scope.go:117] "RemoveContainer" containerID="3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"
Jul 27 23:21:54.771040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.771001    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67"} err="failed to get container status \"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67\": rpc error: code = NotFound desc = could not find container \"3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67\": container with ID starting with 3cc7ef01214e959b44b77040f88b428ec33981df2e6400588cf26e3776179d67 not found: ID does not exist"
Jul 27 23:21:54.771040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.771022    2570 scope.go:117] "RemoveContainer" containerID="ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"
Jul 27 23:21:54.771323 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.771187    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd"} err="failed to get container status \"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd\": rpc error: code = NotFound desc = could not find container \"ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd\": container with ID starting with ce6dae921e80921154f90698583604133211fbdbf36091088567a9ef0d52a6fd not found: ID does not exist"
Jul 27 23:21:54.771323 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.771201    2570 scope.go:117] "RemoveContainer" containerID="f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"
Jul 27 23:21:54.771443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.771392    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a"} err="failed to get container status \"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a\": rpc error: code = NotFound desc = could not find container \"f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a\": container with ID starting with f802e4cd6aeb87b48c8450ff13ca7464e1e00cab7e9352b3448607900263394a not found: ID does not exist"
Jul 27 23:21:54.775337 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.775319    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.778019 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778001    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 23:21:54.778019 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778011    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 23:21:54.778163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778030    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 23:21:54.778163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778033    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 23:21:54.778163 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778097    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 23:21:54.778474 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778456    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 23:21:54.778569 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778531    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 23:21:54.778569 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778533    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 23:21:54.778663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.778647    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g6c2p\""
Jul 27 23:21:54.783507 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.783485    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 23:21:54.784982 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.784965    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:21:54.786947 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.786927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787045 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.786964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787045 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.786982    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787045 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787045 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787041    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787188 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787188 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-kube-api-access-9bxh5\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787188 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787139    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787188 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-config-out\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787302 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787192    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787302 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787227    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-web-config\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787302 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.787415 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.787330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888273 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-kube-api-access-9bxh5\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888273 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888281    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888311    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-config-out\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888333    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-web-config\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888515    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888601    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888680    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888705    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888996 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888729    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.888996 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.888757    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.889588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.889561    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.891741 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.891702    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-web-config\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.891862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.891761    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.891862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.891840    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892172 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892138    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-config-out\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892172 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892166    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892319 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892186    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892319 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892234    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892412 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892331    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892667 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892647    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.892704 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.892675    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.894009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.893985    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:54.895810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:54.895791    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxh5\" (UniqueName: \"kubernetes.io/projected/d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3-kube-api-access-9bxh5\") pod \"alertmanager-main-0\" (UID: \"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:55.086087 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:55.086037    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 23:21:55.222304 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:55.222272    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 23:21:55.225411 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:21:55.225383    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b5d8d8_4640_49dd_a16b_489fd7fa0aa3.slice/crio-22ca55a012423f442054da57ec15d0cf68de6d34547031102da810d4a9c7691f WatchSource:0}: Error finding container 22ca55a012423f442054da57ec15d0cf68de6d34547031102da810d4a9c7691f: Status 404 returned error can't find the container with id 22ca55a012423f442054da57ec15d0cf68de6d34547031102da810d4a9c7691f
Jul 27 23:21:55.520500 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:55.520467    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2749ac-c411-4947-92b6-1050c4bb1746" path="/var/lib/kubelet/pods/0f2749ac-c411-4947-92b6-1050c4bb1746/volumes"
Jul 27 23:21:55.723405 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:55.723344    2570 generic.go:356] "Generic (PLEG): container finished" podID="d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3" containerID="fb02d8d277ecca34afcffff229d7940cd1b330e53d03d1c38d3ba6d17d6f997d" exitCode=0
Jul 27 23:21:55.723567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:55.723415    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerDied","Data":"fb02d8d277ecca34afcffff229d7940cd1b330e53d03d1c38d3ba6d17d6f997d"}
Jul 27 23:21:55.723567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:55.723485    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"22ca55a012423f442054da57ec15d0cf68de6d34547031102da810d4a9c7691f"}
Jul 27 23:21:56.730166 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.730130    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"cb6ee81e87876cd837bb692b8d5f02336eeddf8190f2bb96490107e4a0e3188e"}
Jul 27 23:21:56.730547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.730209    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"22598654ebb6c25557a9cc9907b5b77dc52ccc2639c52372b6b5ca0c60d2dcf7"}
Jul 27 23:21:56.730547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.730246    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"287e1ab2c11a0cc1d5e398e5fb4522314f71dd1126e7e6b7365c76808be03165"}
Jul 27 23:21:56.730547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.730286    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"7dbc4339ae4c3fa600a4c3c6c444a94cae9701b62a8e7039c278c87128856ec5"}
Jul 27 23:21:56.730547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.730322    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"00e7d6976d497749fa1f410a7c8ba5e73dbf79845dde0e69eb0ea9d50a374dc7"}
Jul 27 23:21:56.730547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.730374    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3","Type":"ContainerStarted","Data":"274ff89d609b1e8aa045721469d1da6b172955efaeb448cf89dac20cd39cd6b8"}
Jul 27 23:21:56.763339 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.763283    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7460086280000002 podStartE2EDuration="2.763264597s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.255969ms" isStatefulPod=false podCreationTimestamp="2026-07-27 23:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:21:56.758458437 +0000 UTC m=+341.709801848" watchObservedRunningTime="2026-07-27 23:21:56.763264597 +0000 UTC m=+341.714607983"
Jul 27 23:21:56.998748 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:56.998665    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"]
Jul 27 23:21:57.002307 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.002287    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.004984 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.004954    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-j6f6s\""
Jul 27 23:21:57.005086 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.004960    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 23:21:57.005161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.005112    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 23:21:57.005256 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.005238    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 23:21:57.005306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.005287    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 23:21:57.005376 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.005332    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 23:21:57.009783 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.009754    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 23:21:57.014989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.014968    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"]
Jul 27 23:21:57.106908 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.106874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-federate-client-tls\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.106908 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.106914    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.107145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.106937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-serving-certs-ca-bundle\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.107145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.106954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-metrics-client-ca\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.107145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.107037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.107145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.107091    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhcgz\" (UniqueName: \"kubernetes.io/projected/76418801-f504-450f-9bff-b67a129a62d9-kube-api-access-qhcgz\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.107267 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.107145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-secret-telemeter-client\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.107267 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.107164    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-telemeter-client-tls\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208022 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.207985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208022 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhcgz\" (UniqueName: \"kubernetes.io/projected/76418801-f504-450f-9bff-b67a129a62d9-kube-api-access-qhcgz\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-secret-telemeter-client\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-telemeter-client-tls\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208105    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-federate-client-tls\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-serving-certs-ca-bundle\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.208215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.208168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-metrics-client-ca\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.209081 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.209041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-serving-certs-ca-bundle\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.209267 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.209053    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.209267 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.209086    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76418801-f504-450f-9bff-b67a129a62d9-metrics-client-ca\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.210834 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.210803    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-telemeter-client-tls\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.210940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.210882    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.211184 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.211169    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-federate-client-tls\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.211338 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.211319    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/76418801-f504-450f-9bff-b67a129a62d9-secret-telemeter-client\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.215536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.215512    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhcgz\" (UniqueName: \"kubernetes.io/projected/76418801-f504-450f-9bff-b67a129a62d9-kube-api-access-qhcgz\") pod \"telemeter-client-6cb4fb8fd7-j4r7l\" (UID: \"76418801-f504-450f-9bff-b67a129a62d9\") " pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.311989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.311900    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"
Jul 27 23:21:57.443664 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.443634    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l"]
Jul 27 23:21:57.446133 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:21:57.446102    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76418801_f504_450f_9bff_b67a129a62d9.slice/crio-fbb85fb40a798966fb137e5f05df484df24a5a495fb874f317d0202e8ebeba5c WatchSource:0}: Error finding container fbb85fb40a798966fb137e5f05df484df24a5a495fb874f317d0202e8ebeba5c: Status 404 returned error can't find the container with id fbb85fb40a798966fb137e5f05df484df24a5a495fb874f317d0202e8ebeba5c
Jul 27 23:21:57.448309 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.448294    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:21:57.734977 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:57.734943    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l" event={"ID":"76418801-f504-450f-9bff-b67a129a62d9","Type":"ContainerStarted","Data":"fbb85fb40a798966fb137e5f05df484df24a5a495fb874f317d0202e8ebeba5c"}
Jul 27 23:21:59.746238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:59.746203    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l" event={"ID":"76418801-f504-450f-9bff-b67a129a62d9","Type":"ContainerStarted","Data":"63305716538ce5db812a80aeeafb6533d5b6c42f242f4e2ad995db28de913b98"}
Jul 27 23:21:59.746662 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:59.746260    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l" event={"ID":"76418801-f504-450f-9bff-b67a129a62d9","Type":"ContainerStarted","Data":"87f12e556c1a1721b318ff06fcdd6ce8fa860a51433f5a501c26fbaf3e54525d"}
Jul 27 23:21:59.746662 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:59.746280    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l" event={"ID":"76418801-f504-450f-9bff-b67a129a62d9","Type":"ContainerStarted","Data":"c586998a03ee3a569188f5c97df030abf935ab135a27128253f98ddc33905854"}
Jul 27 23:21:59.768023 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:21:59.767963    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cb4fb8fd7-j4r7l" podStartSLOduration=1.891185108 podStartE2EDuration="3.767944957s" totalImagesPullingTime="1.876759849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:21:59.766124615 +0000 UTC m=+344.717468035" watchObservedRunningTime="2026-07-27 23:21:59.767944957 +0000 UTC m=+344.719288344"
Jul 27 23:24:30.670424 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.670385    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"]
Jul 27 23:24:30.673698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.673678    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.676398 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.676377    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 23:24:30.677668 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.677652    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 23:24:30.677765 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.677690    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lc84s\""
Jul 27 23:24:30.680127 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.680107    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"]
Jul 27 23:24:30.810897 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.810862    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2ae4863-844c-427e-a1ae-537bf1797dd2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ngb5b\" (UID: \"e2ae4863-844c-427e-a1ae-537bf1797dd2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.811102 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.810990    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pth\" (UniqueName: \"kubernetes.io/projected/e2ae4863-844c-427e-a1ae-537bf1797dd2-kube-api-access-v2pth\") pod \"cert-manager-cainjector-559cdd8c87-ngb5b\" (UID: \"e2ae4863-844c-427e-a1ae-537bf1797dd2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.912027 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.911986    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pth\" (UniqueName: \"kubernetes.io/projected/e2ae4863-844c-427e-a1ae-537bf1797dd2-kube-api-access-v2pth\") pod \"cert-manager-cainjector-559cdd8c87-ngb5b\" (UID: \"e2ae4863-844c-427e-a1ae-537bf1797dd2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.912027 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.912036    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2ae4863-844c-427e-a1ae-537bf1797dd2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ngb5b\" (UID: \"e2ae4863-844c-427e-a1ae-537bf1797dd2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.920052 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.920021    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2ae4863-844c-427e-a1ae-537bf1797dd2-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ngb5b\" (UID: \"e2ae4863-844c-427e-a1ae-537bf1797dd2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.920182 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.920147    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pth\" (UniqueName: \"kubernetes.io/projected/e2ae4863-844c-427e-a1ae-537bf1797dd2-kube-api-access-v2pth\") pod \"cert-manager-cainjector-559cdd8c87-ngb5b\" (UID: \"e2ae4863-844c-427e-a1ae-537bf1797dd2\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:30.983164 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:30.983074    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"
Jul 27 23:24:31.109592 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:31.109571    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b"]
Jul 27 23:24:31.111724 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:24:31.111697    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2ae4863_844c_427e_a1ae_537bf1797dd2.slice/crio-b58fab3904b1eb8ea283b9e38d2be151a488eb4aee93a9bda8d3ed1af42d921b WatchSource:0}: Error finding container b58fab3904b1eb8ea283b9e38d2be151a488eb4aee93a9bda8d3ed1af42d921b: Status 404 returned error can't find the container with id b58fab3904b1eb8ea283b9e38d2be151a488eb4aee93a9bda8d3ed1af42d921b
Jul 27 23:24:31.315444 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:31.315347    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b" event={"ID":"e2ae4863-844c-427e-a1ae-537bf1797dd2","Type":"ContainerStarted","Data":"b58fab3904b1eb8ea283b9e38d2be151a488eb4aee93a9bda8d3ed1af42d921b"}
Jul 27 23:24:34.330090 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:34.330055    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b" event={"ID":"e2ae4863-844c-427e-a1ae-537bf1797dd2","Type":"ContainerStarted","Data":"1fe3674a1d0d5a6c897e9e1bd15a8ee4935c97beb165e0a570e52bd5da868414"}
Jul 27 23:24:34.345191 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:34.345132    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ngb5b" podStartSLOduration=1.3501372759999999 podStartE2EDuration="4.345109994s" totalImagesPullingTime="2.994972718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:24:34.34450594 +0000 UTC m=+499.295849332" watchObservedRunningTime="2026-07-27 23:24:34.345109994 +0000 UTC m=+499.296453382"
Jul 27 23:24:41.492226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.492194    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l4xtx"]
Jul 27 23:24:41.495463 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.495446    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.498149 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.498129    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cp555\""
Jul 27 23:24:41.503388 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.503364    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l4xtx"]
Jul 27 23:24:41.603497 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.603457    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/767bbb06-0dd6-4d29-b1f7-1a6b686485e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-l4xtx\" (UID: \"767bbb06-0dd6-4d29-b1f7-1a6b686485e0\") " pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.603497 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.603500    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6q8\" (UniqueName: \"kubernetes.io/projected/767bbb06-0dd6-4d29-b1f7-1a6b686485e0-kube-api-access-dp6q8\") pod \"cert-manager-55b4d954d5-l4xtx\" (UID: \"767bbb06-0dd6-4d29-b1f7-1a6b686485e0\") " pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.705078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.705037    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/767bbb06-0dd6-4d29-b1f7-1a6b686485e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-l4xtx\" (UID: \"767bbb06-0dd6-4d29-b1f7-1a6b686485e0\") " pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.705078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.705078    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6q8\" (UniqueName: \"kubernetes.io/projected/767bbb06-0dd6-4d29-b1f7-1a6b686485e0-kube-api-access-dp6q8\") pod \"cert-manager-55b4d954d5-l4xtx\" (UID: \"767bbb06-0dd6-4d29-b1f7-1a6b686485e0\") " pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.713198 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.713163    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/767bbb06-0dd6-4d29-b1f7-1a6b686485e0-bound-sa-token\") pod \"cert-manager-55b4d954d5-l4xtx\" (UID: \"767bbb06-0dd6-4d29-b1f7-1a6b686485e0\") " pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.713319 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.713231    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6q8\" (UniqueName: \"kubernetes.io/projected/767bbb06-0dd6-4d29-b1f7-1a6b686485e0-kube-api-access-dp6q8\") pod \"cert-manager-55b4d954d5-l4xtx\" (UID: \"767bbb06-0dd6-4d29-b1f7-1a6b686485e0\") " pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.805109 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.805022    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l4xtx"
Jul 27 23:24:41.934449 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:41.934423    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l4xtx"]
Jul 27 23:24:41.936443 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:24:41.936416    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod767bbb06_0dd6_4d29_b1f7_1a6b686485e0.slice/crio-287040c7432a4de09f861ef0a3d19ff8c2a0d5eca26c5a08402105b6ded91f8f WatchSource:0}: Error finding container 287040c7432a4de09f861ef0a3d19ff8c2a0d5eca26c5a08402105b6ded91f8f: Status 404 returned error can't find the container with id 287040c7432a4de09f861ef0a3d19ff8c2a0d5eca26c5a08402105b6ded91f8f
Jul 27 23:24:42.361793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:42.361755    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l4xtx" event={"ID":"767bbb06-0dd6-4d29-b1f7-1a6b686485e0","Type":"ContainerStarted","Data":"6440a510443bfaef610099aa80f7a73ef4ddc73f0dc8fa4a37641aa49399489c"}
Jul 27 23:24:42.361793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:42.361795    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l4xtx" event={"ID":"767bbb06-0dd6-4d29-b1f7-1a6b686485e0","Type":"ContainerStarted","Data":"287040c7432a4de09f861ef0a3d19ff8c2a0d5eca26c5a08402105b6ded91f8f"}
Jul 27 23:24:42.377850 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:24:42.377796    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-l4xtx" podStartSLOduration=1.3777790429999999 podStartE2EDuration="1.377779043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:24:42.377628016 +0000 UTC m=+507.328971416" watchObservedRunningTime="2026-07-27 23:24:42.377779043 +0000 UTC m=+507.329122431"
Jul 27 23:25:08.022177 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.022090    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"]
Jul 27 23:25:08.062414 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.062374    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"]
Jul 27 23:25:08.062600 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.062451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.066834 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.066812    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8t6lg\""
Jul 27 23:25:08.067035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.066815    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 23:25:08.067035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.066815    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 23:25:08.067205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.067011    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 23:25:08.067205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.066853    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 23:25:08.067205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.066839    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 23:25:08.114960 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.114925    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqd5\" (UniqueName: \"kubernetes.io/projected/d4b2d777-5620-4cbf-b73e-67bc58956c4f-kube-api-access-2kqd5\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.115241 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.115217    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4b2d777-5620-4cbf-b73e-67bc58956c4f-metrics-cert\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.115370 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.115254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d4b2d777-5620-4cbf-b73e-67bc58956c4f-manager-config\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.115427 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.115399    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4b2d777-5620-4cbf-b73e-67bc58956c4f-cert\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.215809 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.215774    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4b2d777-5620-4cbf-b73e-67bc58956c4f-cert\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.215986 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.215819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqd5\" (UniqueName: \"kubernetes.io/projected/d4b2d777-5620-4cbf-b73e-67bc58956c4f-kube-api-access-2kqd5\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.215986 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.215874    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4b2d777-5620-4cbf-b73e-67bc58956c4f-metrics-cert\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.215986 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.215900    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d4b2d777-5620-4cbf-b73e-67bc58956c4f-manager-config\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.216811 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.216785    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d4b2d777-5620-4cbf-b73e-67bc58956c4f-manager-config\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.218623 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.218595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4b2d777-5620-4cbf-b73e-67bc58956c4f-metrics-cert\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.218714 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.218635    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4b2d777-5620-4cbf-b73e-67bc58956c4f-cert\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.231973 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.231935    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqd5\" (UniqueName: \"kubernetes.io/projected/d4b2d777-5620-4cbf-b73e-67bc58956c4f-kube-api-access-2kqd5\") pod \"lws-controller-manager-6589476d7c-nnrpx\" (UID: \"d4b2d777-5620-4cbf-b73e-67bc58956c4f\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.373979 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.373880    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:08.510791 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:08.510756    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"]
Jul 27 23:25:08.513726 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:25:08.513700    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b2d777_5620_4cbf_b73e_67bc58956c4f.slice/crio-05923f5e9d9a8789e4a5d7aaeb50a02234e2af667c891f5dbab293657eb76e82 WatchSource:0}: Error finding container 05923f5e9d9a8789e4a5d7aaeb50a02234e2af667c891f5dbab293657eb76e82: Status 404 returned error can't find the container with id 05923f5e9d9a8789e4a5d7aaeb50a02234e2af667c891f5dbab293657eb76e82
Jul 27 23:25:09.072528 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.072490    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"]
Jul 27 23:25:09.075568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.075550    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.079732 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.079700    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 23:25:09.080013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.079888    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 23:25:09.080013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.079966    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 23:25:09.080013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.080009    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 23:25:09.080236 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.080077    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ts86f\""
Jul 27 23:25:09.098925 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.098891    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"]
Jul 27 23:25:09.123545 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.123502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkd4\" (UniqueName: \"kubernetes.io/projected/41b05f88-317b-4659-bc01-44f6d991b868-kube-api-access-kkkd4\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.123545 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.123548    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41b05f88-317b-4659-bc01-44f6d991b868-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.123752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.123582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/41b05f88-317b-4659-bc01-44f6d991b868-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.123752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.123624    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b05f88-317b-4659-bc01-44f6d991b868-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.123752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.123650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/41b05f88-317b-4659-bc01-44f6d991b868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.224135 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.224097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41b05f88-317b-4659-bc01-44f6d991b868-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.224317 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.224155    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/41b05f88-317b-4659-bc01-44f6d991b868-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.224317 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.224183    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b05f88-317b-4659-bc01-44f6d991b868-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.224317 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.224211    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/41b05f88-317b-4659-bc01-44f6d991b868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.224317 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.224312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkd4\" (UniqueName: \"kubernetes.io/projected/41b05f88-317b-4659-bc01-44f6d991b868-kube-api-access-kkkd4\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.224686 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.224647    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/41b05f88-317b-4659-bc01-44f6d991b868-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.226722 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.226699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41b05f88-317b-4659-bc01-44f6d991b868-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.226942 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.226923    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41b05f88-317b-4659-bc01-44f6d991b868-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.226982 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.226935    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/41b05f88-317b-4659-bc01-44f6d991b868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.232326 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.232306    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkd4\" (UniqueName: \"kubernetes.io/projected/41b05f88-317b-4659-bc01-44f6d991b868-kube-api-access-kkkd4\") pod \"opendatahub-operator-controller-manager-9cf867746-ml926\" (UID: \"41b05f88-317b-4659-bc01-44f6d991b868\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.386537 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.386502    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:09.472662 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.472622    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx" event={"ID":"d4b2d777-5620-4cbf-b73e-67bc58956c4f","Type":"ContainerStarted","Data":"05923f5e9d9a8789e4a5d7aaeb50a02234e2af667c891f5dbab293657eb76e82"}
Jul 27 23:25:09.522431 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:09.522410    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"]
Jul 27 23:25:09.524568 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:25:09.524545    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b05f88_317b_4659_bc01_44f6d991b868.slice/crio-54c342ccff1e25c4b5e07d7a3ba97dd873ec41df2fb171f543c0501dfb20eedb WatchSource:0}: Error finding container 54c342ccff1e25c4b5e07d7a3ba97dd873ec41df2fb171f543c0501dfb20eedb: Status 404 returned error can't find the container with id 54c342ccff1e25c4b5e07d7a3ba97dd873ec41df2fb171f543c0501dfb20eedb
Jul 27 23:25:10.480571 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:10.480494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926" event={"ID":"41b05f88-317b-4659-bc01-44f6d991b868","Type":"ContainerStarted","Data":"54c342ccff1e25c4b5e07d7a3ba97dd873ec41df2fb171f543c0501dfb20eedb"}
Jul 27 23:25:13.495486 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:13.495389    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx" event={"ID":"d4b2d777-5620-4cbf-b73e-67bc58956c4f","Type":"ContainerStarted","Data":"dc70c18693f1ff39d751cd8eb07bb9ac9d527a4660bbbfb55e35f11374dfdf84"}
Jul 27 23:25:13.495486 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:13.495452    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:13.496827 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:13.496804    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926" event={"ID":"41b05f88-317b-4659-bc01-44f6d991b868","Type":"ContainerStarted","Data":"f4b68500aa4664144f2ba2c4e5f537bd2a4e0069f39be22053fdb5d002ac2eaa"}
Jul 27 23:25:13.511922 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:13.511760    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx" podStartSLOduration=1.799294427 podStartE2EDuration="6.511743633s" totalImagesPullingTime="4.712449206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:13.511671529 +0000 UTC m=+538.463014916" watchObservedRunningTime="2026-07-27 23:25:13.511743633 +0000 UTC m=+538.463087023"
Jul 27 23:25:14.503062 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:14.503024    2570 generic.go:356] "Generic (PLEG): container finished" podID="41b05f88-317b-4659-bc01-44f6d991b868" containerID="f4b68500aa4664144f2ba2c4e5f537bd2a4e0069f39be22053fdb5d002ac2eaa" exitCode=0
Jul 27 23:25:14.503435 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:14.503102    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926" event={"ID":"41b05f88-317b-4659-bc01-44f6d991b868","Type":"ContainerDied","Data":"f4b68500aa4664144f2ba2c4e5f537bd2a4e0069f39be22053fdb5d002ac2eaa"}
Jul 27 23:25:15.508878 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:15.508847    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926" event={"ID":"41b05f88-317b-4659-bc01-44f6d991b868","Type":"ContainerStarted","Data":"07ad91c8bd741236ef736d541a89646609eeb3b842ba1c83ecfb7b7d3d83a605"}
Jul 27 23:25:15.509280 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:15.508981    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:15.532129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:15.532067    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926" podStartSLOduration=1.350410253 podStartE2EDuration="6.532046971s" totalImagesPullingTime="4.03654638s" totalInitContainerRuntime="1.145090338s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:15.530030162 +0000 UTC m=+540.481373561" watchObservedRunningTime="2026-07-27 23:25:15.532046971 +0000 UTC m=+540.483390359"
Jul 27 23:25:24.505958 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:24.505927    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-nnrpx"
Jul 27 23:25:26.516067 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:26.516038    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-ml926"
Jul 27 23:25:36.844816 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.844782    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"]
Jul 27 23:25:36.853021 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.852999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.855729 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855703    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 23:25:36.855862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855769    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 23:25:36.855862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855783    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 23:25:36.855862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855807    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 23:25:36.855862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855818    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9ljqj\""
Jul 27 23:25:36.856049 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855928    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 23:25:36.856049 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.855945    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 23:25:36.861882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.861860    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"]
Jul 27 23:25:36.969717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969684    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.969912 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969736    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.969912 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.969912 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969792    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.969912 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.970071 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13739f21-ca0e-46c1-93b0-081b976f9ea5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:36.970071 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:36.969950    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw82j\" (UniqueName: \"kubernetes.io/projected/13739f21-ca0e-46c1-93b0-081b976f9ea5-kube-api-access-fw82j\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13739f21-ca0e-46c1-93b0-081b976f9ea5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071124    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw82j\" (UniqueName: \"kubernetes.io/projected/13739f21-ca0e-46c1-93b0-081b976f9ea5-kube-api-access-fw82j\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071408 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071408 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071172    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071408 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071189    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071408 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071408 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071291    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.071849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.071812    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.073642 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.073616    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/13739f21-ca0e-46c1-93b0-081b976f9ea5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.073861 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.073841    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.073953 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.073939    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.074044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.074023    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.085811 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.085789    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw82j\" (UniqueName: \"kubernetes.io/projected/13739f21-ca0e-46c1-93b0-081b976f9ea5-kube-api-access-fw82j\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.085974 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.085952    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13739f21-ca0e-46c1-93b0-081b976f9ea5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zzbzc\" (UID: \"13739f21-ca0e-46c1-93b0-081b976f9ea5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.163535 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.163507    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:37.323065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.322995    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"]
Jul 27 23:25:37.325701 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:25:37.325675    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13739f21_ca0e_46c1_93b0_081b976f9ea5.slice/crio-1868979fa750a5f8357aff947709fffd4a0859e919fe6c5ef2cca575c1beaa26 WatchSource:0}: Error finding container 1868979fa750a5f8357aff947709fffd4a0859e919fe6c5ef2cca575c1beaa26: Status 404 returned error can't find the container with id 1868979fa750a5f8357aff947709fffd4a0859e919fe6c5ef2cca575c1beaa26
Jul 27 23:25:37.605509 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:37.605422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc" event={"ID":"13739f21-ca0e-46c1-93b0-081b976f9ea5","Type":"ContainerStarted","Data":"1868979fa750a5f8357aff947709fffd4a0859e919fe6c5ef2cca575c1beaa26"}
Jul 27 23:25:39.786438 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:39.786390    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:25:39.786708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:39.786482    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:25:40.624051 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:40.624012    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc" event={"ID":"13739f21-ca0e-46c1-93b0-081b976f9ea5","Type":"ContainerStarted","Data":"a098759519ac23d960f0c7d8fc2c05d1ea2066a9d56a5b8ba84055c8a02c5f66"}
Jul 27 23:25:40.624259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:40.624210    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:25:40.626857 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:40.626565    2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-zzbzc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 23:25:40.626857 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:40.626618    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc" podUID="13739f21-ca0e-46c1-93b0-081b976f9ea5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:25:40.654797 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:40.654745    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc" podStartSLOduration=2.196725562 podStartE2EDuration="4.654731965s" totalImagesPullingTime="2.458006403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:40.653222015 +0000 UTC m=+565.604565423" watchObservedRunningTime="2026-07-27 23:25:40.654731965 +0000 UTC m=+565.606075352"
Jul 27 23:25:41.628891 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:25:41.628859    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zzbzc"
Jul 27 23:26:15.394833 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:26:15.394805    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:26:15.396238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:26:15.396220    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:27:57.291065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.291028    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:27:57.294263 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.294244    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.296729 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.296702    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-27gkk\""
Jul 27 23:27:57.296844 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.296816    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 23:27:57.297951 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.297934    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 23:27:57.298043 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.297950    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 23:27:57.301723 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.301325    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:27:57.387870 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.387823    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:27:57.394515 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.394484    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99xg\" (UniqueName: \"kubernetes.io/projected/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-kube-api-access-c99xg\") pod \"limitador-limitador-64c94fd64f-6dbwg\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.394655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.394530    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-config-file\") pod \"limitador-limitador-64c94fd64f-6dbwg\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.495463 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.495426    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99xg\" (UniqueName: \"kubernetes.io/projected/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-kube-api-access-c99xg\") pod \"limitador-limitador-64c94fd64f-6dbwg\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.495634 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.495476    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-config-file\") pod \"limitador-limitador-64c94fd64f-6dbwg\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.496092 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.496071    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-config-file\") pod \"limitador-limitador-64c94fd64f-6dbwg\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.506527 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.506492    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99xg\" (UniqueName: \"kubernetes.io/projected/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-kube-api-access-c99xg\") pod \"limitador-limitador-64c94fd64f-6dbwg\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.605272 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.605183    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:27:57.737314 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.737294    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:27:57.740137 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:27:57.740112    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7903b5d_90f0_41b3_90c5_7ddea4ce7c3c.slice/crio-52a3d9d4a9ad3dec0089c778be1d67be7eb9f6d1d8ef628f5b133d3a81491acd WatchSource:0}: Error finding container 52a3d9d4a9ad3dec0089c778be1d67be7eb9f6d1d8ef628f5b133d3a81491acd: Status 404 returned error can't find the container with id 52a3d9d4a9ad3dec0089c778be1d67be7eb9f6d1d8ef628f5b133d3a81491acd
Jul 27 23:27:57.742083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:57.742068    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:27:58.223735 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:27:58.223701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg" event={"ID":"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c","Type":"ContainerStarted","Data":"52a3d9d4a9ad3dec0089c778be1d67be7eb9f6d1d8ef628f5b133d3a81491acd"}
Jul 27 23:28:01.240543 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:01.240499    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg" event={"ID":"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c","Type":"ContainerStarted","Data":"80f80f3cd57d5e95de16562f60ef6cc82cf7e1c9802046a3357b1254d1b07a7e"}
Jul 27 23:28:01.240967 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:01.240648    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:28:01.257988 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:01.257938    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg" podStartSLOduration=1.400510198 podStartE2EDuration="4.257922189s" totalImagesPullingTime="2.857411991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:01.256397294 +0000 UTC m=+706.207740672" watchObservedRunningTime="2026-07-27 23:28:01.257922189 +0000 UTC m=+706.209265576"
Jul 27 23:28:12.245719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:12.245639    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:28:13.870999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:13.870958    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:28:13.871472 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:13.871144    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg" podUID="b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" containerName="limitador" containerID="cri-o://80f80f3cd57d5e95de16562f60ef6cc82cf7e1c9802046a3357b1254d1b07a7e" gracePeriod=30
Jul 27 23:28:14.301116 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.301086    2570 generic.go:356] "Generic (PLEG): container finished" podID="b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" containerID="80f80f3cd57d5e95de16562f60ef6cc82cf7e1c9802046a3357b1254d1b07a7e" exitCode=0
Jul 27 23:28:14.301244 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.301162    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg" event={"ID":"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c","Type":"ContainerDied","Data":"80f80f3cd57d5e95de16562f60ef6cc82cf7e1c9802046a3357b1254d1b07a7e"}
Jul 27 23:28:14.416719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.416695    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:28:14.453440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.453347    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c99xg\" (UniqueName: \"kubernetes.io/projected/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-kube-api-access-c99xg\") pod \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") "
Jul 27 23:28:14.453596 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.453454    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-config-file\") pod \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\" (UID: \"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c\") "
Jul 27 23:28:14.453829 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.453798    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-config-file" (OuterVolumeSpecName: "config-file") pod "b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" (UID: "b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:28:14.455781 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.455758    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-kube-api-access-c99xg" (OuterVolumeSpecName: "kube-api-access-c99xg") pod "b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" (UID: "b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c"). InnerVolumeSpecName "kube-api-access-c99xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:28:14.554952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.554912    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99xg\" (UniqueName: \"kubernetes.io/projected/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-kube-api-access-c99xg\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:28:14.554952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:14.554948    2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c-config-file\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:28:15.307494 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:15.307461    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg" event={"ID":"b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c","Type":"ContainerDied","Data":"52a3d9d4a9ad3dec0089c778be1d67be7eb9f6d1d8ef628f5b133d3a81491acd"}
Jul 27 23:28:15.307494 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:15.307490    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"
Jul 27 23:28:15.307995 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:15.307556    2570 scope.go:117] "RemoveContainer" containerID="80f80f3cd57d5e95de16562f60ef6cc82cf7e1c9802046a3357b1254d1b07a7e"
Jul 27 23:28:15.328211 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:15.328184    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:28:15.330661 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:15.330633    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6dbwg"]
Jul 27 23:28:15.521414 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:15.521383    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" path="/var/lib/kubelet/pods/b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c/volumes"
Jul 27 23:28:37.993689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:37.993655    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f556dcc9c-2hxc4"]
Jul 27 23:28:37.994066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:37.993966    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" containerName="limitador"
Jul 27 23:28:37.994066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:37.993977    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" containerName="limitador"
Jul 27 23:28:37.994066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:37.994039    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7903b5d-90f0-41b3-90c5-7ddea4ce7c3c" containerName="limitador"
Jul 27 23:28:38.002514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.002492    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.006161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.006136    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 23:28:38.006667 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.006646    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5vmcw\""
Jul 27 23:28:38.006932 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.006915    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 23:28:38.007390 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.007335    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f556dcc9c-2hxc4"]
Jul 27 23:28:38.145562 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.145527    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-559685848-sj8ck"]
Jul 27 23:28:38.149174 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.149158    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.158448 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.158418    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-559685848-sj8ck"]
Jul 27 23:28:38.162519 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.162497    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcn58\" (UniqueName: \"kubernetes.io/projected/468958f6-596a-4f87-b164-6fd1827864a9-kube-api-access-bcn58\") pod \"maas-controller-5f556dcc9c-2hxc4\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") " pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.162645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.162534    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468958f6-596a-4f87-b164-6fd1827864a9-webhook-cert\") pod \"maas-controller-5f556dcc9c-2hxc4\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") " pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.263440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.263291    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-webhook-cert\") pod \"maas-controller-559685848-sj8ck\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") " pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.263440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.263330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2h9\" (UniqueName: \"kubernetes.io/projected/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-kube-api-access-lm2h9\") pod \"maas-controller-559685848-sj8ck\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") " pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.263440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.263405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcn58\" (UniqueName: \"kubernetes.io/projected/468958f6-596a-4f87-b164-6fd1827864a9-kube-api-access-bcn58\") pod \"maas-controller-5f556dcc9c-2hxc4\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") " pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.263687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.263487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468958f6-596a-4f87-b164-6fd1827864a9-webhook-cert\") pod \"maas-controller-5f556dcc9c-2hxc4\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") " pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.266054 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.266033    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468958f6-596a-4f87-b164-6fd1827864a9-webhook-cert\") pod \"maas-controller-5f556dcc9c-2hxc4\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") " pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.272643 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.272614    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcn58\" (UniqueName: \"kubernetes.io/projected/468958f6-596a-4f87-b164-6fd1827864a9-kube-api-access-bcn58\") pod \"maas-controller-5f556dcc9c-2hxc4\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") " pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.314387 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.314344    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:38.364965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.364649    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-webhook-cert\") pod \"maas-controller-559685848-sj8ck\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") " pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.364965 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.364705    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2h9\" (UniqueName: \"kubernetes.io/projected/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-kube-api-access-lm2h9\") pod \"maas-controller-559685848-sj8ck\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") " pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.368099 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.368069    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-webhook-cert\") pod \"maas-controller-559685848-sj8ck\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") " pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.373207 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.373172    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2h9\" (UniqueName: \"kubernetes.io/projected/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-kube-api-access-lm2h9\") pod \"maas-controller-559685848-sj8ck\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") " pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.449389 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.449341    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f556dcc9c-2hxc4"]
Jul 27 23:28:38.451332 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:28:38.451307    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468958f6_596a_4f87_b164_6fd1827864a9.slice/crio-c4c8adb9bd44d693f807b2ed4ad2f67f612000865e9f23bc9538aaf34f483a7e WatchSource:0}: Error finding container c4c8adb9bd44d693f807b2ed4ad2f67f612000865e9f23bc9538aaf34f483a7e: Status 404 returned error can't find the container with id c4c8adb9bd44d693f807b2ed4ad2f67f612000865e9f23bc9538aaf34f483a7e
Jul 27 23:28:38.459788 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.459767    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:38.586592 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:38.586571    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-559685848-sj8ck"]
Jul 27 23:28:38.589096 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:28:38.589077    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a98b11d_c2e0_4924_b5b1_37a230e0bec7.slice/crio-7d0afb0de25d23e4171a296b0acd460386bbb785ea7083fe3d07be0134e32c09 WatchSource:0}: Error finding container 7d0afb0de25d23e4171a296b0acd460386bbb785ea7083fe3d07be0134e32c09: Status 404 returned error can't find the container with id 7d0afb0de25d23e4171a296b0acd460386bbb785ea7083fe3d07be0134e32c09
Jul 27 23:28:39.414933 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:39.414890    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-sj8ck" event={"ID":"1a98b11d-c2e0-4924-b5b1-37a230e0bec7","Type":"ContainerStarted","Data":"7d0afb0de25d23e4171a296b0acd460386bbb785ea7083fe3d07be0134e32c09"}
Jul 27 23:28:39.416730 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:39.416698    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4" event={"ID":"468958f6-596a-4f87-b164-6fd1827864a9","Type":"ContainerStarted","Data":"c4c8adb9bd44d693f807b2ed4ad2f67f612000865e9f23bc9538aaf34f483a7e"}
Jul 27 23:28:41.427435 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:41.427398    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-sj8ck" event={"ID":"1a98b11d-c2e0-4924-b5b1-37a230e0bec7","Type":"ContainerStarted","Data":"476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b"}
Jul 27 23:28:41.427917 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:41.427587    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:41.429645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:41.429616    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4" event={"ID":"468958f6-596a-4f87-b164-6fd1827864a9","Type":"ContainerStarted","Data":"935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b"}
Jul 27 23:28:41.429799 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:41.429691    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:41.447156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:41.447112    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-559685848-sj8ck" podStartSLOduration=0.823408795 podStartE2EDuration="3.447097418s" totalImagesPullingTime="2.623688623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:41.445320897 +0000 UTC m=+746.396664308" watchObservedRunningTime="2026-07-27 23:28:41.447097418 +0000 UTC m=+746.398440805"
Jul 27 23:28:41.463069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:41.462982    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4" podStartSLOduration=1.694897964 podStartE2EDuration="4.462954868s" totalImagesPullingTime="2.768056904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:41.462123776 +0000 UTC m=+746.413467163" watchObservedRunningTime="2026-07-27 23:28:41.462954868 +0000 UTC m=+746.414298254"
Jul 27 23:28:52.436933 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.436901    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:28:52.437493 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.436973    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:52.493796 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.493758    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f556dcc9c-2hxc4"]
Jul 27 23:28:52.494015 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.493972    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4" podUID="468958f6-596a-4f87-b164-6fd1827864a9" containerName="manager" containerID="cri-o://935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b" gracePeriod=30
Jul 27 23:28:52.736707 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.736680    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:52.796847 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.796806    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcn58\" (UniqueName: \"kubernetes.io/projected/468958f6-596a-4f87-b164-6fd1827864a9-kube-api-access-bcn58\") pod \"468958f6-596a-4f87-b164-6fd1827864a9\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") "
Jul 27 23:28:52.797238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.796858    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468958f6-596a-4f87-b164-6fd1827864a9-webhook-cert\") pod \"468958f6-596a-4f87-b164-6fd1827864a9\" (UID: \"468958f6-596a-4f87-b164-6fd1827864a9\") "
Jul 27 23:28:52.799130 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.799108    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468958f6-596a-4f87-b164-6fd1827864a9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "468958f6-596a-4f87-b164-6fd1827864a9" (UID: "468958f6-596a-4f87-b164-6fd1827864a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:28:52.799224 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.799207    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468958f6-596a-4f87-b164-6fd1827864a9-kube-api-access-bcn58" (OuterVolumeSpecName: "kube-api-access-bcn58") pod "468958f6-596a-4f87-b164-6fd1827864a9" (UID: "468958f6-596a-4f87-b164-6fd1827864a9"). InnerVolumeSpecName "kube-api-access-bcn58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:28:52.898293 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.898254    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcn58\" (UniqueName: \"kubernetes.io/projected/468958f6-596a-4f87-b164-6fd1827864a9-kube-api-access-bcn58\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:28:52.898293 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:52.898286    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/468958f6-596a-4f87-b164-6fd1827864a9-webhook-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:28:53.496036 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.496002    2570 generic.go:356] "Generic (PLEG): container finished" podID="468958f6-596a-4f87-b164-6fd1827864a9" containerID="935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b" exitCode=0
Jul 27 23:28:53.496514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.496043    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4"
Jul 27 23:28:53.496514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.496067    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4" event={"ID":"468958f6-596a-4f87-b164-6fd1827864a9","Type":"ContainerDied","Data":"935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b"}
Jul 27 23:28:53.496514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.496109    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f556dcc9c-2hxc4" event={"ID":"468958f6-596a-4f87-b164-6fd1827864a9","Type":"ContainerDied","Data":"c4c8adb9bd44d693f807b2ed4ad2f67f612000865e9f23bc9538aaf34f483a7e"}
Jul 27 23:28:53.496514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.496130    2570 scope.go:117] "RemoveContainer" containerID="935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b"
Jul 27 23:28:53.505135 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.505116    2570 scope.go:117] "RemoveContainer" containerID="935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b"
Jul 27 23:28:53.505414 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:28:53.505379    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b\": container with ID starting with 935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b not found: ID does not exist" containerID="935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b"
Jul 27 23:28:53.505414 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.505404    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b"} err="failed to get container status \"935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b\": rpc error: code = NotFound desc = could not find container \"935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b\": container with ID starting with 935fc1ae4ad4aa6b36719a78cfbe19469cfd1d4dbf14951226355199c1ba183b not found: ID does not exist"
Jul 27 23:28:53.521764 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.521737    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f556dcc9c-2hxc4"]
Jul 27 23:28:53.522529 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:53.522511    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f556dcc9c-2hxc4"]
Jul 27 23:28:55.522900 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:28:55.522867    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="468958f6-596a-4f87-b164-6fd1827864a9" path="/var/lib/kubelet/pods/468958f6-596a-4f87-b164-6fd1827864a9/volumes"
Jul 27 23:29:11.599777 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.599744    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"]
Jul 27 23:29:11.600254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.600160    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="468958f6-596a-4f87-b164-6fd1827864a9" containerName="manager"
Jul 27 23:29:11.600254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.600175    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="468958f6-596a-4f87-b164-6fd1827864a9" containerName="manager"
Jul 27 23:29:11.600254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.600249    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="468958f6-596a-4f87-b164-6fd1827864a9" containerName="manager"
Jul 27 23:29:11.603179 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.603162    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.605877 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.605859    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 23:29:11.605992 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.605881    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7chpm\""
Jul 27 23:29:11.605992 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.605917    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 23:29:11.607194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.607174    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 23:29:11.613483 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.613438    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"]
Jul 27 23:29:11.758766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.758732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ec049-a86a-41b2-8fac-d5ecf4add937-maas-api-tls\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.758936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.758768    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d69l\" (UniqueName: \"kubernetes.io/projected/d94ec049-a86a-41b2-8fac-d5ecf4add937-kube-api-access-2d69l\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.758936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.758849    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ec049-a86a-41b2-8fac-d5ecf4add937-openshift-service-ca\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.859901 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.859806    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ec049-a86a-41b2-8fac-d5ecf4add937-openshift-service-ca\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.860056 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.859922    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ec049-a86a-41b2-8fac-d5ecf4add937-maas-api-tls\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.860056 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.859944    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d69l\" (UniqueName: \"kubernetes.io/projected/d94ec049-a86a-41b2-8fac-d5ecf4add937-kube-api-access-2d69l\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.860126 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:29:11.860059    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 27 23:29:11.860163 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:29:11.860131    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d94ec049-a86a-41b2-8fac-d5ecf4add937-maas-api-tls podName:d94ec049-a86a-41b2-8fac-d5ecf4add937 nodeName:}" failed. No retries permitted until 2026-07-27 23:29:12.360110994 +0000 UTC m=+777.311454365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d94ec049-a86a-41b2-8fac-d5ecf4add937-maas-api-tls") pod "maas-api-6cbbdc4ff7-lzqqx" (UID: "d94ec049-a86a-41b2-8fac-d5ecf4add937") : secret "maas-api-serving-cert" not found
Jul 27 23:29:11.860494 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.860471    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94ec049-a86a-41b2-8fac-d5ecf4add937-openshift-service-ca\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:11.871991 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:11.871969    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d69l\" (UniqueName: \"kubernetes.io/projected/d94ec049-a86a-41b2-8fac-d5ecf4add937-kube-api-access-2d69l\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:12.182012 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.181977    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"]
Jul 27 23:29:12.185454 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.185434    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.188526 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.188508    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kqt9q\""
Jul 27 23:29:12.188614 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.188511    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 23:29:12.196927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.196902    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"]
Jul 27 23:29:12.227252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.227216    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mcpnn"]
Jul 27 23:29:12.230390 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.230374    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.239895 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.239859    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mcpnn"]
Jul 27 23:29:12.263028 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.262972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76621e53-64f0-4420-96bd-178173721996-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2cf9t\" (UID: \"76621e53-64f0-4420-96bd-178173721996\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.263194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.263062    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bxs\" (UniqueName: \"kubernetes.io/projected/76621e53-64f0-4420-96bd-178173721996-kube-api-access-75bxs\") pod \"payload-pre-processing-687dbb6cf6-2cf9t\" (UID: \"76621e53-64f0-4420-96bd-178173721996\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.364428 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.364386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pks\" (UniqueName: \"kubernetes.io/projected/b0d6a3ab-abed-40dd-a462-a521cbdef40e-kube-api-access-44pks\") pod \"payload-processing-5cc6cd444b-mcpnn\" (UID: \"b0d6a3ab-abed-40dd-a462-a521cbdef40e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.364626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.364447    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76621e53-64f0-4420-96bd-178173721996-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2cf9t\" (UID: \"76621e53-64f0-4420-96bd-178173721996\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.364626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.364533    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bxs\" (UniqueName: \"kubernetes.io/projected/76621e53-64f0-4420-96bd-178173721996-kube-api-access-75bxs\") pod \"payload-pre-processing-687dbb6cf6-2cf9t\" (UID: \"76621e53-64f0-4420-96bd-178173721996\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.364626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.364600    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d6a3ab-abed-40dd-a462-a521cbdef40e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mcpnn\" (UID: \"b0d6a3ab-abed-40dd-a462-a521cbdef40e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.364794 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.364739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ec049-a86a-41b2-8fac-d5ecf4add937-maas-api-tls\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:12.365212 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.365185    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76621e53-64f0-4420-96bd-178173721996-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-2cf9t\" (UID: \"76621e53-64f0-4420-96bd-178173721996\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.367575 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.367549    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d94ec049-a86a-41b2-8fac-d5ecf4add937-maas-api-tls\") pod \"maas-api-6cbbdc4ff7-lzqqx\" (UID: \"d94ec049-a86a-41b2-8fac-d5ecf4add937\") " pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:12.372505 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.372479    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bxs\" (UniqueName: \"kubernetes.io/projected/76621e53-64f0-4420-96bd-178173721996-kube-api-access-75bxs\") pod \"payload-pre-processing-687dbb6cf6-2cf9t\" (UID: \"76621e53-64f0-4420-96bd-178173721996\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.465953 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.465851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pks\" (UniqueName: \"kubernetes.io/projected/b0d6a3ab-abed-40dd-a462-a521cbdef40e-kube-api-access-44pks\") pod \"payload-processing-5cc6cd444b-mcpnn\" (UID: \"b0d6a3ab-abed-40dd-a462-a521cbdef40e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.465953 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.465936    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d6a3ab-abed-40dd-a462-a521cbdef40e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mcpnn\" (UID: \"b0d6a3ab-abed-40dd-a462-a521cbdef40e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.466619 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.466601    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d6a3ab-abed-40dd-a462-a521cbdef40e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-mcpnn\" (UID: \"b0d6a3ab-abed-40dd-a462-a521cbdef40e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.474340 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.474311    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pks\" (UniqueName: \"kubernetes.io/projected/b0d6a3ab-abed-40dd-a462-a521cbdef40e-kube-api-access-44pks\") pod \"payload-processing-5cc6cd444b-mcpnn\" (UID: \"b0d6a3ab-abed-40dd-a462-a521cbdef40e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.497047 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.497016    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:12.514250 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.514222    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:12.546572 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.546535    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:12.657120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.657093    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"]
Jul 27 23:29:12.659000 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:29:12.658966    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76621e53_64f0_4420_96bd_178173721996.slice/crio-be06df3e245dd6c7260f10b2ad20f90722434dd8b6ae199b54b2c11de9db19af WatchSource:0}: Error finding container be06df3e245dd6c7260f10b2ad20f90722434dd8b6ae199b54b2c11de9db19af: Status 404 returned error can't find the container with id be06df3e245dd6c7260f10b2ad20f90722434dd8b6ae199b54b2c11de9db19af
Jul 27 23:29:12.688985 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.688962    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"]
Jul 27 23:29:12.691913 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:29:12.691884    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94ec049_a86a_41b2_8fac_d5ecf4add937.slice/crio-ce13d7b6c5fae22d753899e6771f62c7c5237c35d929400cdd7f3f99955dad49 WatchSource:0}: Error finding container ce13d7b6c5fae22d753899e6771f62c7c5237c35d929400cdd7f3f99955dad49: Status 404 returned error can't find the container with id ce13d7b6c5fae22d753899e6771f62c7c5237c35d929400cdd7f3f99955dad49
Jul 27 23:29:12.713777 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:12.713752    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-mcpnn"]
Jul 27 23:29:12.715307 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:29:12.715282    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d6a3ab_abed_40dd_a462_a521cbdef40e.slice/crio-5096d7cd2265f030577fc54f03b5dcd8089c39931138d47bfe5d9c57d304d165 WatchSource:0}: Error finding container 5096d7cd2265f030577fc54f03b5dcd8089c39931138d47bfe5d9c57d304d165: Status 404 returned error can't find the container with id 5096d7cd2265f030577fc54f03b5dcd8089c39931138d47bfe5d9c57d304d165
Jul 27 23:29:13.596734 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:13.596653    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t" event={"ID":"76621e53-64f0-4420-96bd-178173721996","Type":"ContainerStarted","Data":"be06df3e245dd6c7260f10b2ad20f90722434dd8b6ae199b54b2c11de9db19af"}
Jul 27 23:29:13.599926 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:13.599848    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx" event={"ID":"d94ec049-a86a-41b2-8fac-d5ecf4add937","Type":"ContainerStarted","Data":"ce13d7b6c5fae22d753899e6771f62c7c5237c35d929400cdd7f3f99955dad49"}
Jul 27 23:29:13.602810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:13.602732    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn" event={"ID":"b0d6a3ab-abed-40dd-a462-a521cbdef40e","Type":"ContainerStarted","Data":"5096d7cd2265f030577fc54f03b5dcd8089c39931138d47bfe5d9c57d304d165"}
Jul 27 23:29:15.613906 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.613873    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t" event={"ID":"76621e53-64f0-4420-96bd-178173721996","Type":"ContainerStarted","Data":"54d9b377ce8195776b4c95f46cbb068f3a4cb9da8ecbcac9310bf0af81144281"}
Jul 27 23:29:15.614316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.613978    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:15.615385 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.615341    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx" event={"ID":"d94ec049-a86a-41b2-8fac-d5ecf4add937","Type":"ContainerStarted","Data":"7f3fd384d40bd83aab5de2503303d3cca8b796c634f440f6637ceb934004d75c"}
Jul 27 23:29:15.615504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.615446    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:15.616722 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.616700    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn" event={"ID":"b0d6a3ab-abed-40dd-a462-a521cbdef40e","Type":"ContainerStarted","Data":"b845f8638028000e0a9b259e58a66631972ed1ef94ca10d84ad775c775968f9a"}
Jul 27 23:29:15.616793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.616763    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:15.629653 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.629608    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t" podStartSLOduration=1.403868259 podStartE2EDuration="3.629594943s" totalImagesPullingTime="2.225726684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:29:15.627898912 +0000 UTC m=+780.579242323" watchObservedRunningTime="2026-07-27 23:29:15.629594943 +0000 UTC m=+780.580938330"
Jul 27 23:29:15.643298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.643256    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx" podStartSLOduration=2.456013219 podStartE2EDuration="4.643242122s" totalImagesPullingTime="2.187228903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:29:15.641859586 +0000 UTC m=+780.593202970" watchObservedRunningTime="2026-07-27 23:29:15.643242122 +0000 UTC m=+780.594585509"
Jul 27 23:29:15.656333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:15.656296    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn" podStartSLOduration=1.492463385 podStartE2EDuration="3.656283128s" totalImagesPullingTime="2.163819743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:29:15.655759893 +0000 UTC m=+780.607103297" watchObservedRunningTime="2026-07-27 23:29:15.656283128 +0000 UTC m=+780.607626514"
Jul 27 23:29:21.627142 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:21.627112    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cbbdc4ff7-lzqqx"
Jul 27 23:29:26.624124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:26.624096    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-2cf9t"
Jul 27 23:29:26.624728 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:26.624176    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-mcpnn"
Jul 27 23:29:32.650897 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.650866    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"]
Jul 27 23:29:32.658327 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.658307    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.661132 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.661108    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dz654\""
Jul 27 23:29:32.661132 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.661108    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 27 23:29:32.661319 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.661150    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 23:29:32.661319 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.661150    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 23:29:32.661824 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.661806    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"]
Jul 27 23:29:32.752793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.752755    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6f8763e-5292-41e7-b70e-c041e7a1abbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.752978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.752823    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.752978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.752882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.752978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.752943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.752978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.752971    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwjd\" (UniqueName: \"kubernetes.io/projected/e6f8763e-5292-41e7-b70e-c041e7a1abbe-kube-api-access-skwjd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.753145 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.752997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.853672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.853631    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.853672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.853678    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.853919 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.853702    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.853919 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.853720    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwjd\" (UniqueName: \"kubernetes.io/projected/e6f8763e-5292-41e7-b70e-c041e7a1abbe-kube-api-access-skwjd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.853919 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.853748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.853919 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.853793    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6f8763e-5292-41e7-b70e-c041e7a1abbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.854125 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.854083    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.854204 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.854179    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.854371 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.854337    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.856215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.856188    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6f8763e-5292-41e7-b70e-c041e7a1abbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.856343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.856286    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6f8763e-5292-41e7-b70e-c041e7a1abbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.861642 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.861619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwjd\" (UniqueName: \"kubernetes.io/projected/e6f8763e-5292-41e7-b70e-c041e7a1abbe-kube-api-access-skwjd\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx\" (UID: \"e6f8763e-5292-41e7-b70e-c041e7a1abbe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:32.969921 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:32.969835    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:33.112628 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:33.112601    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"]
Jul 27 23:29:33.114373 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:29:33.114332    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f8763e_5292_41e7_b70e_c041e7a1abbe.slice/crio-b6ae843d97c6b58baf0c5a62820b39bb5a757cd7f446f5b852e40ff7279ad465 WatchSource:0}: Error finding container b6ae843d97c6b58baf0c5a62820b39bb5a757cd7f446f5b852e40ff7279ad465: Status 404 returned error can't find the container with id b6ae843d97c6b58baf0c5a62820b39bb5a757cd7f446f5b852e40ff7279ad465
Jul 27 23:29:33.704266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:33.704223    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx" event={"ID":"e6f8763e-5292-41e7-b70e-c041e7a1abbe","Type":"ContainerStarted","Data":"b6ae843d97c6b58baf0c5a62820b39bb5a757cd7f446f5b852e40ff7279ad465"}
Jul 27 23:29:34.710737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:34.710692    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx" event={"ID":"e6f8763e-5292-41e7-b70e-c041e7a1abbe","Type":"ContainerStarted","Data":"c62a82d3a2820bc3854f647694a2841e55970809479adbe4a7b69bacaed931c2"}
Jul 27 23:29:34.728233 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:34.728184    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx" podStartSLOduration=1.7572145670000001 podStartE2EDuration="2.728169207s" totalImagesPullingTime="970.95464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:29:34.72706899 +0000 UTC m=+799.678412378" watchObservedRunningTime="2026-07-27 23:29:34.728169207 +0000 UTC m=+799.679512593"
Jul 27 23:29:42.640822 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.640745    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"]
Jul 27 23:29:42.649649 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.649624    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.651668 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.651643    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"]
Jul 27 23:29:42.652331 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.652307    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 27 23:29:42.743626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.743593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.743793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.743659    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhhx\" (UniqueName: \"kubernetes.io/projected/99b79ebe-4109-4a1c-8140-544be46f29c1-kube-api-access-qhhhx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.743793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.743719    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.743793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.743746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.743951 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.743815    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.743951 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.743869    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b79ebe-4109-4a1c-8140-544be46f29c1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.844981 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.844942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.844981 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.844983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845000    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845128    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b79ebe-4109-4a1c-8140-544be46f29c1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845165    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845401 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845228    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhhx\" (UniqueName: \"kubernetes.io/projected/99b79ebe-4109-4a1c-8140-544be46f29c1-kube-api-access-qhhhx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845463 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845418    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845463 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845451    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.845577 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.845557    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.847546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.847516    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99b79ebe-4109-4a1c-8140-544be46f29c1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.847912 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.847889    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b79ebe-4109-4a1c-8140-544be46f29c1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.853330 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.853306    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhhx\" (UniqueName: \"kubernetes.io/projected/99b79ebe-4109-4a1c-8140-544be46f29c1-kube-api-access-qhhhx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kznlr\" (UID: \"99b79ebe-4109-4a1c-8140-544be46f29c1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.960820 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.960739    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:42.970000 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.969976    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:42.970122 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.970034    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:42.984297 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:42.984272    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:43.101113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:43.101086    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"]
Jul 27 23:29:43.103104 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:29:43.103071    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b79ebe_4109_4a1c_8140_544be46f29c1.slice/crio-07584f97404946dd6afc2cac42f79aa95824d9094de0bd8b286dfe21534b25b1 WatchSource:0}: Error finding container 07584f97404946dd6afc2cac42f79aa95824d9094de0bd8b286dfe21534b25b1: Status 404 returned error can't find the container with id 07584f97404946dd6afc2cac42f79aa95824d9094de0bd8b286dfe21534b25b1
Jul 27 23:29:43.753876 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:43.753840    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr" event={"ID":"99b79ebe-4109-4a1c-8140-544be46f29c1","Type":"ContainerStarted","Data":"f6f91f73c49f247ac864f636b21d36fec5b5899951fe0cf13dff456750e73dcf"}
Jul 27 23:29:43.754340 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:43.753895    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr" event={"ID":"99b79ebe-4109-4a1c-8140-544be46f29c1","Type":"ContainerStarted","Data":"07584f97404946dd6afc2cac42f79aa95824d9094de0bd8b286dfe21534b25b1"}
Jul 27 23:29:43.765204 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:43.765179    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx"
Jul 27 23:29:43.776765 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:43.776718    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr" podStartSLOduration=1.614715862 podStartE2EDuration="1.776701396s" totalImagesPullingTime="161.985534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:29:43.775228359 +0000 UTC m=+808.726571774" watchObservedRunningTime="2026-07-27 23:29:43.776701396 +0000 UTC m=+808.728044859"
Jul 27 23:29:52.960988 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:52.960951    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:52.961443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:52.961015    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:52.974297 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:52.974274    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:29:53.810332 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:29:53.810296    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kznlr"
Jul 27 23:31:15.424814 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:31:15.424746    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:31:15.424814 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:31:15.424746    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:32:16.682194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.682154    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-559685848-sj8ck"]
Jul 27 23:32:16.682787 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.682480    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-559685848-sj8ck" podUID="1a98b11d-c2e0-4924-b5b1-37a230e0bec7" containerName="manager" containerID="cri-o://476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b" gracePeriod=30
Jul 27 23:32:16.929514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.929488    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:32:16.929987 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.929974    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-webhook-cert\") pod \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") "
Jul 27 23:32:16.930033 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.930007    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm2h9\" (UniqueName: \"kubernetes.io/projected/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-kube-api-access-lm2h9\") pod \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\" (UID: \"1a98b11d-c2e0-4924-b5b1-37a230e0bec7\") "
Jul 27 23:32:16.932191 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.932160    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-kube-api-access-lm2h9" (OuterVolumeSpecName: "kube-api-access-lm2h9") pod "1a98b11d-c2e0-4924-b5b1-37a230e0bec7" (UID: "1a98b11d-c2e0-4924-b5b1-37a230e0bec7"). InnerVolumeSpecName "kube-api-access-lm2h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:32:16.932191 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:16.932165    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1a98b11d-c2e0-4924-b5b1-37a230e0bec7" (UID: "1a98b11d-c2e0-4924-b5b1-37a230e0bec7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:32:17.030813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.030781    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm2h9\" (UniqueName: \"kubernetes.io/projected/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-kube-api-access-lm2h9\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:32:17.030813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.030811    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a98b11d-c2e0-4924-b5b1-37a230e0bec7-webhook-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:32:17.459258 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.459220    2570 generic.go:356] "Generic (PLEG): container finished" podID="1a98b11d-c2e0-4924-b5b1-37a230e0bec7" containerID="476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b" exitCode=0
Jul 27 23:32:17.459467 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.459269    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-sj8ck"
Jul 27 23:32:17.459467 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.459279    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-sj8ck" event={"ID":"1a98b11d-c2e0-4924-b5b1-37a230e0bec7","Type":"ContainerDied","Data":"476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b"}
Jul 27 23:32:17.459467 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.459323    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-sj8ck" event={"ID":"1a98b11d-c2e0-4924-b5b1-37a230e0bec7","Type":"ContainerDied","Data":"7d0afb0de25d23e4171a296b0acd460386bbb785ea7083fe3d07be0134e32c09"}
Jul 27 23:32:17.459467 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.459346    2570 scope.go:117] "RemoveContainer" containerID="476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b"
Jul 27 23:32:17.468841 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.468821    2570 scope.go:117] "RemoveContainer" containerID="476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b"
Jul 27 23:32:17.469151 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:32:17.469135    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b\": container with ID starting with 476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b not found: ID does not exist" containerID="476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b"
Jul 27 23:32:17.469193 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.469159    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b"} err="failed to get container status \"476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b\": rpc error: code = NotFound desc = could not find container \"476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b\": container with ID starting with 476b519744a11056bc569e4184a184e200f2d42de8516a909a2566aec925e68b not found: ID does not exist"
Jul 27 23:32:17.479243 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.479210    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-559685848-sj8ck"]
Jul 27 23:32:17.482584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.482541    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-559685848-sj8ck"]
Jul 27 23:32:17.521996 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:32:17.521962    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a98b11d-c2e0-4924-b5b1-37a230e0bec7" path="/var/lib/kubelet/pods/1a98b11d-c2e0-4924-b5b1-37a230e0bec7/volumes"
Jul 27 23:33:18.275726 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.275677    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-559685848-gm2nf"]
Jul 27 23:33:18.276226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.276194    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a98b11d-c2e0-4924-b5b1-37a230e0bec7" containerName="manager"
Jul 27 23:33:18.276226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.276214    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a98b11d-c2e0-4924-b5b1-37a230e0bec7" containerName="manager"
Jul 27 23:33:18.276342 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.276318    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a98b11d-c2e0-4924-b5b1-37a230e0bec7" containerName="manager"
Jul 27 23:33:18.279612 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.279592    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.282333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.282313    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 23:33:18.283515 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.283497    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5vmcw\""
Jul 27 23:33:18.283689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.283664    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 23:33:18.288755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.288734    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-559685848-gm2nf"]
Jul 27 23:33:18.373120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.373076    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/375d56e1-5dde-47df-8be7-b1ca3b21e9de-webhook-cert\") pod \"maas-controller-559685848-gm2nf\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") " pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.373295 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.373147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvlwt\" (UniqueName: \"kubernetes.io/projected/375d56e1-5dde-47df-8be7-b1ca3b21e9de-kube-api-access-lvlwt\") pod \"maas-controller-559685848-gm2nf\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") " pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.474620 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.474576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvlwt\" (UniqueName: \"kubernetes.io/projected/375d56e1-5dde-47df-8be7-b1ca3b21e9de-kube-api-access-lvlwt\") pod \"maas-controller-559685848-gm2nf\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") " pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.474786 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.474663    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/375d56e1-5dde-47df-8be7-b1ca3b21e9de-webhook-cert\") pod \"maas-controller-559685848-gm2nf\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") " pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.477205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.477185    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/375d56e1-5dde-47df-8be7-b1ca3b21e9de-webhook-cert\") pod \"maas-controller-559685848-gm2nf\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") " pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.482739 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.482715    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvlwt\" (UniqueName: \"kubernetes.io/projected/375d56e1-5dde-47df-8be7-b1ca3b21e9de-kube-api-access-lvlwt\") pod \"maas-controller-559685848-gm2nf\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") " pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.591737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.591639    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:18.943584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.943552    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-559685848-gm2nf"]
Jul 27 23:33:18.944289 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:18.944265    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:33:19.737598 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:19.737512    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-gm2nf" event={"ID":"375d56e1-5dde-47df-8be7-b1ca3b21e9de","Type":"ContainerStarted","Data":"ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6"}
Jul 27 23:33:19.737598 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:19.737563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-gm2nf" event={"ID":"375d56e1-5dde-47df-8be7-b1ca3b21e9de","Type":"ContainerStarted","Data":"49ced7f8091b7c47039f9fdd6162a4cd73d33fc6be146826f924bf23028021be"}
Jul 27 23:33:19.737977 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:19.737608    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:19.754687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:19.754635    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-559685848-gm2nf" podStartSLOduration=1.296504217 podStartE2EDuration="1.754618188s" totalImagesPullingTime="458.113971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:33:19.752339296 +0000 UTC m=+1024.703682684" watchObservedRunningTime="2026-07-27 23:33:19.754618188 +0000 UTC m=+1024.705961583"
Jul 27 23:33:30.745214 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:30.745174    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:35.363955 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.363917    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-559685848-gm2nf"]
Jul 27 23:33:35.364465 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.364197    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-559685848-gm2nf" podUID="375d56e1-5dde-47df-8be7-b1ca3b21e9de" containerName="manager" containerID="cri-o://ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6" gracePeriod=30
Jul 27 23:33:35.614389 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.614316    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:35.738703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.738668    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvlwt\" (UniqueName: \"kubernetes.io/projected/375d56e1-5dde-47df-8be7-b1ca3b21e9de-kube-api-access-lvlwt\") pod \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") "
Jul 27 23:33:35.738899 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.738781    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/375d56e1-5dde-47df-8be7-b1ca3b21e9de-webhook-cert\") pod \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\" (UID: \"375d56e1-5dde-47df-8be7-b1ca3b21e9de\") "
Jul 27 23:33:35.741045 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.741014    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375d56e1-5dde-47df-8be7-b1ca3b21e9de-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "375d56e1-5dde-47df-8be7-b1ca3b21e9de" (UID: "375d56e1-5dde-47df-8be7-b1ca3b21e9de"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:33:35.741166 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.741079    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375d56e1-5dde-47df-8be7-b1ca3b21e9de-kube-api-access-lvlwt" (OuterVolumeSpecName: "kube-api-access-lvlwt") pod "375d56e1-5dde-47df-8be7-b1ca3b21e9de" (UID: "375d56e1-5dde-47df-8be7-b1ca3b21e9de"). InnerVolumeSpecName "kube-api-access-lvlwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:33:35.815393 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.815333    2570 generic.go:356] "Generic (PLEG): container finished" podID="375d56e1-5dde-47df-8be7-b1ca3b21e9de" containerID="ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6" exitCode=0
Jul 27 23:33:35.815613 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.815416    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-gm2nf"
Jul 27 23:33:35.815613 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.815426    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-gm2nf" event={"ID":"375d56e1-5dde-47df-8be7-b1ca3b21e9de","Type":"ContainerDied","Data":"ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6"}
Jul 27 23:33:35.815613 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.815486    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-gm2nf" event={"ID":"375d56e1-5dde-47df-8be7-b1ca3b21e9de","Type":"ContainerDied","Data":"49ced7f8091b7c47039f9fdd6162a4cd73d33fc6be146826f924bf23028021be"}
Jul 27 23:33:35.815613 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.815519    2570 scope.go:117] "RemoveContainer" containerID="ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6"
Jul 27 23:33:35.824868 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.824845    2570 scope.go:117] "RemoveContainer" containerID="ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6"
Jul 27 23:33:35.825131 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:33:35.825109    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6\": container with ID starting with ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6 not found: ID does not exist" containerID="ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6"
Jul 27 23:33:35.825202 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.825144    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6"} err="failed to get container status \"ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6\": rpc error: code = NotFound desc = could not find container \"ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6\": container with ID starting with ebc9ab756cb17cac32838341d3d503ca62e5cf94b1a0046a891f8e3b61ca4fb6 not found: ID does not exist"
Jul 27 23:33:35.836756 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.836726    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-559685848-gm2nf"]
Jul 27 23:33:35.839650 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.839621    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-559685848-gm2nf"]
Jul 27 23:33:35.839966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.839949    2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/375d56e1-5dde-47df-8be7-b1ca3b21e9de-webhook-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:33:35.840012 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:35.839971    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvlwt\" (UniqueName: \"kubernetes.io/projected/375d56e1-5dde-47df-8be7-b1ca3b21e9de-kube-api-access-lvlwt\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:33:36.603119 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.603082    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-559685848-vmr9g"]
Jul 27 23:33:36.603524 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.603468    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="375d56e1-5dde-47df-8be7-b1ca3b21e9de" containerName="manager"
Jul 27 23:33:36.603524 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.603483    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="375d56e1-5dde-47df-8be7-b1ca3b21e9de" containerName="manager"
Jul 27 23:33:36.603596 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.603560    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="375d56e1-5dde-47df-8be7-b1ca3b21e9de" containerName="manager"
Jul 27 23:33:36.607575 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.607559    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.611313 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.611292    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 23:33:36.611852 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.611342    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 23:33:36.611852 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.611399    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5vmcw\""
Jul 27 23:33:36.614152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.614132    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-559685848-vmr9g"]
Jul 27 23:33:36.747553 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.747515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pddl\" (UniqueName: \"kubernetes.io/projected/68e30622-2f81-4f69-9c2e-4780d5b50551-kube-api-access-8pddl\") pod \"maas-controller-559685848-vmr9g\" (UID: \"68e30622-2f81-4f69-9c2e-4780d5b50551\") " pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.747553 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.747555    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68e30622-2f81-4f69-9c2e-4780d5b50551-webhook-cert\") pod \"maas-controller-559685848-vmr9g\" (UID: \"68e30622-2f81-4f69-9c2e-4780d5b50551\") " pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.848681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.848645    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pddl\" (UniqueName: \"kubernetes.io/projected/68e30622-2f81-4f69-9c2e-4780d5b50551-kube-api-access-8pddl\") pod \"maas-controller-559685848-vmr9g\" (UID: \"68e30622-2f81-4f69-9c2e-4780d5b50551\") " pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.848681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.848686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68e30622-2f81-4f69-9c2e-4780d5b50551-webhook-cert\") pod \"maas-controller-559685848-vmr9g\" (UID: \"68e30622-2f81-4f69-9c2e-4780d5b50551\") " pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.851280 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.851260    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68e30622-2f81-4f69-9c2e-4780d5b50551-webhook-cert\") pod \"maas-controller-559685848-vmr9g\" (UID: \"68e30622-2f81-4f69-9c2e-4780d5b50551\") " pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.856634 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.856571    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pddl\" (UniqueName: \"kubernetes.io/projected/68e30622-2f81-4f69-9c2e-4780d5b50551-kube-api-access-8pddl\") pod \"maas-controller-559685848-vmr9g\" (UID: \"68e30622-2f81-4f69-9c2e-4780d5b50551\") " pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:36.920248 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:36.920219    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:37.053832 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:37.053806    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-559685848-vmr9g"]
Jul 27 23:33:37.055907 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:33:37.055882    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e30622_2f81_4f69_9c2e_4780d5b50551.slice/crio-6476305bd77f45ef8aee0f74aba4043c257e9440c5a8b6d0a6914621c30aeabf WatchSource:0}: Error finding container 6476305bd77f45ef8aee0f74aba4043c257e9440c5a8b6d0a6914621c30aeabf: Status 404 returned error can't find the container with id 6476305bd77f45ef8aee0f74aba4043c257e9440c5a8b6d0a6914621c30aeabf
Jul 27 23:33:37.527387 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:37.527324    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="375d56e1-5dde-47df-8be7-b1ca3b21e9de" path="/var/lib/kubelet/pods/375d56e1-5dde-47df-8be7-b1ca3b21e9de/volumes"
Jul 27 23:33:37.829201 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:37.829109    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-vmr9g" event={"ID":"68e30622-2f81-4f69-9c2e-4780d5b50551","Type":"ContainerStarted","Data":"7e79ad1ea966da9ca33974ce0504aa7a67871bfd59357cc4ec159e212233f644"}
Jul 27 23:33:37.829201 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:37.829164    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-559685848-vmr9g" event={"ID":"68e30622-2f81-4f69-9c2e-4780d5b50551","Type":"ContainerStarted","Data":"6476305bd77f45ef8aee0f74aba4043c257e9440c5a8b6d0a6914621c30aeabf"}
Jul 27 23:33:37.829672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:37.829264    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:33:37.846627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:37.846574    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-559685848-vmr9g" podStartSLOduration=1.484715632 podStartE2EDuration="1.846558333s" totalImagesPullingTime="361.842701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:33:37.845191569 +0000 UTC m=+1042.796534979" watchObservedRunningTime="2026-07-27 23:33:37.846558333 +0000 UTC m=+1042.797901720"
Jul 27 23:33:48.835751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:33:48.835718    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-559685848-vmr9g"
Jul 27 23:36:15.452148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:36:15.452119    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:36:15.453698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:36:15.453674    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:41:15.484291 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:41:15.484261    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:41:15.485651 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:41:15.485630    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:42:17.725701 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.725668    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"]
Jul 27 23:42:17.729553 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.729529    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.731999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.731980    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3dfbadf5-kserve-self-signed-certs\""
Jul 27 23:42:17.737204 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.736902    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"]
Jul 27 23:42:17.824428 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.824382    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tls-certs\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.824599 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.824440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-home\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.824599 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.824498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tmp-dir\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.824599 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.824560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-model-cache\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.824599 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.824598    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-dshm\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.824733 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.824632    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzh5\" (UniqueName: \"kubernetes.io/projected/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-kube-api-access-9mzh5\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.925928 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.925888    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tmp-dir\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.925928 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.925933    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-model-cache\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926169 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.925956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-dshm\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926169 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.926089    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzh5\" (UniqueName: \"kubernetes.io/projected/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-kube-api-access-9mzh5\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926169 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.926156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tls-certs\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.926186    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-home\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.926258    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tmp-dir\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926472 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.926420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-model-cache\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.926565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.926543    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-home\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.928300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.928271    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-dshm\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.928647 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.928632    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tls-certs\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:17.933905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:17.933882    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzh5\" (UniqueName: \"kubernetes.io/projected/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-kube-api-access-9mzh5\") pod \"e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") " pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:18.040416 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:18.040284    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:18.179254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:18.179210    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"]
Jul 27 23:42:18.181596 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:42:18.181566    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5b2246_ba6d_4b50_8c77_8889a32cb0d6.slice/crio-cd897d25a67d182b658f962aa203d2a75fd50c2c1c739b629229067dd8419a52 WatchSource:0}: Error finding container cd897d25a67d182b658f962aa203d2a75fd50c2c1c739b629229067dd8419a52: Status 404 returned error can't find the container with id cd897d25a67d182b658f962aa203d2a75fd50c2c1c739b629229067dd8419a52
Jul 27 23:42:18.214238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:18.214197    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl" event={"ID":"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6","Type":"ContainerStarted","Data":"cd897d25a67d182b658f962aa203d2a75fd50c2c1c739b629229067dd8419a52"}
Jul 27 23:42:19.221122 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:19.221083    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl" event={"ID":"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6","Type":"ContainerStarted","Data":"514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58"}
Jul 27 23:42:19.243216 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:19.243164    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl" podStartSLOduration=2.243144239 podStartE2EDuration="2.243144239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:42:19.242599427 +0000 UTC m=+1564.193942810" watchObservedRunningTime="2026-07-27 23:42:19.243144239 +0000 UTC m=+1564.194487628"
Jul 27 23:42:28.040698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:28.040656    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:28.040698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:28.040712    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:28.053525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:28.053499    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:28.275169 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:28.275136    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:42:46.830984 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:42:46.830952    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"]
Jul 27 23:43:01.853520 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:01.853477    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl" podUID="ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" containerName="main" containerID="cri-o://514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58" gracePeriod=45
Jul 27 23:43:02.108321 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.108264    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:43:02.133808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.133773    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-dshm\") pod \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") "
Jul 27 23:43:02.133808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.133809    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tls-certs\") pod \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") "
Jul 27 23:43:02.134004 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.133830    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-model-cache\") pod \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") "
Jul 27 23:43:02.134004 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.133915    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-home\") pod \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") "
Jul 27 23:43:02.134004 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.133939    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tmp-dir\") pod \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") "
Jul 27 23:43:02.134105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.134006    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mzh5\" (UniqueName: \"kubernetes.io/projected/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-kube-api-access-9mzh5\") pod \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\" (UID: \"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6\") "
Jul 27 23:43:02.134191 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.134142    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" (UID: "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:02.134275 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.134228    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-home" (OuterVolumeSpecName: "home") pod "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" (UID: "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:02.134333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.134288    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" (UID: "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:02.134417 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.134341    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-model-cache\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:43:02.134417 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.134385    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-home\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:43:02.136197 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.136172    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-dshm" (OuterVolumeSpecName: "dshm") pod "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" (UID: "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:43:02.136821 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.136798    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" (UID: "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:43:02.136901 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.136819    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-kube-api-access-9mzh5" (OuterVolumeSpecName: "kube-api-access-9mzh5") pod "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" (UID: "ef5b2246-ba6d-4b50-8c77-8889a32cb0d6"). InnerVolumeSpecName "kube-api-access-9mzh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:43:02.235813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.235780    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tmp-dir\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:43:02.235813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.235809    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzh5\" (UniqueName: \"kubernetes.io/projected/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-kube-api-access-9mzh5\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:43:02.235813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.235818    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-dshm\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:43:02.235813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.235826    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6-tls-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:43:02.427431 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.427396    2570 generic.go:356] "Generic (PLEG): container finished" podID="ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" containerID="514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58" exitCode=0
Jul 27 23:43:02.427588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.427448    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl" event={"ID":"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6","Type":"ContainerDied","Data":"514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58"}
Jul 27 23:43:02.427588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.427480    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl" event={"ID":"ef5b2246-ba6d-4b50-8c77-8889a32cb0d6","Type":"ContainerDied","Data":"cd897d25a67d182b658f962aa203d2a75fd50c2c1c739b629229067dd8419a52"}
Jul 27 23:43:02.427588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.427498    2570 scope.go:117] "RemoveContainer" containerID="514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58"
Jul 27 23:43:02.427588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.427448    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"
Jul 27 23:43:02.437382 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.437341    2570 scope.go:117] "RemoveContainer" containerID="514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58"
Jul 27 23:43:02.437642 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:43:02.437624    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58\": container with ID starting with 514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58 not found: ID does not exist" containerID="514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58"
Jul 27 23:43:02.437705 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.437654    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58"} err="failed to get container status \"514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58\": rpc error: code = NotFound desc = could not find container \"514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58\": container with ID starting with 514bef573156d72316ad16bc1ebcbb0110168cb86c10c8aa44329b25150dcf58 not found: ID does not exist"
Jul 27 23:43:02.449034 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.448994    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"]
Jul 27 23:43:02.450611 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:02.450590    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3dfbadf5-kserve-5df9f7bddc-wv9wl"]
Jul 27 23:43:03.521614 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:43:03.521580    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" path="/var/lib/kubelet/pods/ef5b2246-ba6d-4b50-8c77-8889a32cb0d6/volumes"
Jul 27 23:45:00.130808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.130730    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"]
Jul 27 23:45:00.133175 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.131090    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" containerName="main"
Jul 27 23:45:00.133175 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.131102    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" containerName="main"
Jul 27 23:45:00.133175 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.131170    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5b2246-ba6d-4b50-8c77-8889a32cb0d6" containerName="main"
Jul 27 23:45:00.134076 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.134062    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"
Jul 27 23:45:00.136605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.136586    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:45:00.143448 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.143429    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"
Jul 27 23:45:00.147494 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.147468    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"]
Jul 27 23:45:00.280808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.280760    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"]
Jul 27 23:45:00.283832 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:45:00.283807    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95646315_1948_47e6_933d_10572c1cfa25.slice/crio-2534ff3fbc21f26031a087d06d5476fc899755ea7be7b00aea586cda751e1955 WatchSource:0}: Error finding container 2534ff3fbc21f26031a087d06d5476fc899755ea7be7b00aea586cda751e1955: Status 404 returned error can't find the container with id 2534ff3fbc21f26031a087d06d5476fc899755ea7be7b00aea586cda751e1955
Jul 27 23:45:00.285726 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.285710    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:45:00.985541 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:00.985508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8" event={"ID":"95646315-1948-47e6-933d-10572c1cfa25","Type":"ContainerStarted","Data":"2534ff3fbc21f26031a087d06d5476fc899755ea7be7b00aea586cda751e1955"}
Jul 27 23:45:02.997398 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:02.997287    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8" event={"ID":"95646315-1948-47e6-933d-10572c1cfa25","Type":"ContainerStarted","Data":"0d94d3773f68f3ec1c2c1e5a83e976ed7df415c98887c19703839115d8080b24"}
Jul 27 23:45:03.014936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:03.014886    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8" podStartSLOduration=0.597587832 podStartE2EDuration="3.014869343s" totalImagesPullingTime="2.417281511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:45:03.013412726 +0000 UTC m=+1727.964756113" watchObservedRunningTime="2026-07-27 23:45:03.014869343 +0000 UTC m=+1727.966212729"
Jul 27 23:45:04.003837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:04.003805    2570 generic.go:356] "Generic (PLEG): container finished" podID="95646315-1948-47e6-933d-10572c1cfa25" containerID="0d94d3773f68f3ec1c2c1e5a83e976ed7df415c98887c19703839115d8080b24" exitCode=0
Jul 27 23:45:04.004206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:04.003900    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8" event={"ID":"95646315-1948-47e6-933d-10572c1cfa25","Type":"ContainerDied","Data":"0d94d3773f68f3ec1c2c1e5a83e976ed7df415c98887c19703839115d8080b24"}
Jul 27 23:45:05.141349 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:05.141323    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"
Jul 27 23:45:06.013923 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:06.013898    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8"
Jul 27 23:45:06.014099 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:06.013895    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753265-hltt8" event={"ID":"95646315-1948-47e6-933d-10572c1cfa25","Type":"ContainerDied","Data":"2534ff3fbc21f26031a087d06d5476fc899755ea7be7b00aea586cda751e1955"}
Jul 27 23:45:06.014099 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:45:06.014018    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2534ff3fbc21f26031a087d06d5476fc899755ea7be7b00aea586cda751e1955"
Jul 27 23:46:15.513956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:15.513928    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:46:15.515564 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:15.515540    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:46:29.419252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.419214    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"]
Jul 27 23:46:29.419896 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.419875    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95646315-1948-47e6-933d-10572c1cfa25" containerName="cleanup"
Jul 27 23:46:29.419990 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.419898    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="95646315-1948-47e6-933d-10572c1cfa25" containerName="cleanup"
Jul 27 23:46:29.420046 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.420025    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="95646315-1948-47e6-933d-10572c1cfa25" containerName="cleanup"
Jul 27 23:46:29.423021 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.422999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.425560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.425541    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-17ce21b6-serving-cert\""
Jul 27 23:46:29.434172 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.434144    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"]
Jul 27 23:46:29.488793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.488753    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxgc\" (UniqueName: \"kubernetes.io/projected/c4fb04a7-210d-42e2-bf70-deed865c4563-kube-api-access-5xxgc\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.489002 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.488803    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fb04a7-210d-42e2-bf70-deed865c4563-maas-api-tls\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.489002 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.488919    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb04a7-210d-42e2-bf70-deed865c4563-openshift-service-ca\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.589497 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.589459    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb04a7-210d-42e2-bf70-deed865c4563-openshift-service-ca\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.589705 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.589529    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxgc\" (UniqueName: \"kubernetes.io/projected/c4fb04a7-210d-42e2-bf70-deed865c4563-kube-api-access-5xxgc\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.589705 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.589557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fb04a7-210d-42e2-bf70-deed865c4563-maas-api-tls\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.590226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.590199    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb04a7-210d-42e2-bf70-deed865c4563-openshift-service-ca\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.592093 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.592064    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fb04a7-210d-42e2-bf70-deed865c4563-maas-api-tls\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.597182 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.597156    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxgc\" (UniqueName: \"kubernetes.io/projected/c4fb04a7-210d-42e2-bf70-deed865c4563-kube-api-access-5xxgc\") pod \"maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.734563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.734454    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:29.888972 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:29.888947    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"]
Jul 27 23:46:29.890697 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:46:29.890672    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fb04a7_210d_42e2_bf70_deed865c4563.slice/crio-56fa1357e2042fc36d454798c50bfdb6c3488b55a40ccbffdfb9005a9bdb9a41 WatchSource:0}: Error finding container 56fa1357e2042fc36d454798c50bfdb6c3488b55a40ccbffdfb9005a9bdb9a41: Status 404 returned error can't find the container with id 56fa1357e2042fc36d454798c50bfdb6c3488b55a40ccbffdfb9005a9bdb9a41
Jul 27 23:46:30.078696 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.078615    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"]
Jul 27 23:46:30.083211 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.083194    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.086179 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.086155    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-17ce21b6-dockercfg-5gjmh\""
Jul 27 23:46:30.086304 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.086155    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-17ce21b6\""
Jul 27 23:46:30.093181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.093155    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"]
Jul 27 23:46:30.127678 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.127643    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"]
Jul 27 23:46:30.131288 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.131270    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.143143 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.143113    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"]
Jul 27 23:46:30.195107 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.195074    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv4nl\" (UniqueName: \"kubernetes.io/projected/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-kube-api-access-bv4nl\") pod \"payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") " pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.195276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.195115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbqzv\" (UniqueName: \"kubernetes.io/projected/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-kube-api-access-mbqzv\") pod \"payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.195276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.195228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-plugins-config-volume\") pod \"payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") " pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.195276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.195260    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.296269 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.296231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv4nl\" (UniqueName: \"kubernetes.io/projected/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-kube-api-access-bv4nl\") pod \"payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") " pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.296269 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.296269    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbqzv\" (UniqueName: \"kubernetes.io/projected/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-kube-api-access-mbqzv\") pod \"payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.296556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.296325    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-plugins-config-volume\") pod \"payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") " pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.296556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.296344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.297150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.297110    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.297150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.297143    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-plugins-config-volume\") pod \"payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") " pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.303920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.303895    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbqzv\" (UniqueName: \"kubernetes.io/projected/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-kube-api-access-mbqzv\") pod \"payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.304018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.303994    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv4nl\" (UniqueName: \"kubernetes.io/projected/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-kube-api-access-bv4nl\") pod \"payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") " pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.393967 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.393939    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:30.401412 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.401386    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p" event={"ID":"c4fb04a7-210d-42e2-bf70-deed865c4563","Type":"ContainerStarted","Data":"56fa1357e2042fc36d454798c50bfdb6c3488b55a40ccbffdfb9005a9bdb9a41"}
Jul 27 23:46:30.444567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.444102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:30.538282 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.538253    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"]
Jul 27 23:46:30.539450 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:46:30.539225    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca52a04f_a999_44b8_a3ff_d0fd4dee2edc.slice/crio-406d2fb16975cdbe28618cf74b9324fda546b1b047168d597f6b9291c091b1b6 WatchSource:0}: Error finding container 406d2fb16975cdbe28618cf74b9324fda546b1b047168d597f6b9291c091b1b6: Status 404 returned error can't find the container with id 406d2fb16975cdbe28618cf74b9324fda546b1b047168d597f6b9291c091b1b6
Jul 27 23:46:30.603903 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:30.603880    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"]
Jul 27 23:46:30.605728 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:46:30.605697    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9744860_4bfa_4f6e_b1b4_e5785b0534cb.slice/crio-26ee61f6cdc7c0d6271d83e55ffc696ce09d0f8e87dea9883ea7add595514227 WatchSource:0}: Error finding container 26ee61f6cdc7c0d6271d83e55ffc696ce09d0f8e87dea9883ea7add595514227: Status 404 returned error can't find the container with id 26ee61f6cdc7c0d6271d83e55ffc696ce09d0f8e87dea9883ea7add595514227
Jul 27 23:46:31.411348 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.411310    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p" event={"ID":"c4fb04a7-210d-42e2-bf70-deed865c4563","Type":"ContainerStarted","Data":"7246067ac29c0a377f709a09a0cbb2b993751cc5eb9736f27b48549f75ff99c2"}
Jul 27 23:46:31.411597 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.411511    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:31.412966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.412937    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6" event={"ID":"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc","Type":"ContainerStarted","Data":"f83bce18744f2d2a03dd55c9af074009d2c36e3413973e9e8243bf73c72743cd"}
Jul 27 23:46:31.413096 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.412983    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6" event={"ID":"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc","Type":"ContainerStarted","Data":"406d2fb16975cdbe28618cf74b9324fda546b1b047168d597f6b9291c091b1b6"}
Jul 27 23:46:31.413096 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.413015    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:31.414308 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.414289    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq" event={"ID":"f9744860-4bfa-4f6e-b1b4-e5785b0534cb","Type":"ContainerStarted","Data":"619013fa2de3d38974b0b6cc89e9d1665ac1982ffcb3429e3ac81c8558e975da"}
Jul 27 23:46:31.414407 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.414331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq" event={"ID":"f9744860-4bfa-4f6e-b1b4-e5785b0534cb","Type":"ContainerStarted","Data":"26ee61f6cdc7c0d6271d83e55ffc696ce09d0f8e87dea9883ea7add595514227"}
Jul 27 23:46:31.414407 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.414366    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:31.429708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.429668    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p" podStartSLOduration=1.833267639 podStartE2EDuration="2.429653554s" totalImagesPullingTime="596.385915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:31.427668467 +0000 UTC m=+1816.379011864" watchObservedRunningTime="2026-07-27 23:46:31.429653554 +0000 UTC m=+1816.380996941"
Jul 27 23:46:31.443806 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.443764    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq" podStartSLOduration=1.443750262 podStartE2EDuration="1.443750262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:31.441748561 +0000 UTC m=+1816.393091967" watchObservedRunningTime="2026-07-27 23:46:31.443750262 +0000 UTC m=+1816.395093649"
Jul 27 23:46:31.457418 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:31.457370    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6" podStartSLOduration=1.457338365 podStartE2EDuration="1.457338365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:46:31.455878669 +0000 UTC m=+1816.407222078" watchObservedRunningTime="2026-07-27 23:46:31.457338365 +0000 UTC m=+1816.408681751"
Jul 27 23:46:37.426396 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:37.426346    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:46:42.422222 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:42.422195    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:46:42.422711 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:42.422260    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:46:54.922324 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:54.922287    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"]
Jul 27 23:46:54.925950 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:54.925928    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:54.928489 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:54.928465    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:46:54.932952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:54.932923    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"]
Jul 27 23:46:55.024215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:55.024164    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f328b-8c2e-4759-943a-764c8e77f63d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4\" (UID: \"fd3f328b-8c2e-4759-943a-764c8e77f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:55.125522 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:55.125488    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f328b-8c2e-4759-943a-764c8e77f63d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4\" (UID: \"fd3f328b-8c2e-4759-943a-764c8e77f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:55.126119 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:55.126099    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f328b-8c2e-4759-943a-764c8e77f63d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4\" (UID: \"fd3f328b-8c2e-4759-943a-764c8e77f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:55.239516 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:55.239430    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:55.380896 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:55.380874    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"]
Jul 27 23:46:55.383707 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:46:55.383675    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3f328b_8c2e_4759_943a_764c8e77f63d.slice/crio-22dcb9d758d5e8402e9c6a7981e6c0fe9cd538900d1d9db7d88fae1efb660dce WatchSource:0}: Error finding container 22dcb9d758d5e8402e9c6a7981e6c0fe9cd538900d1d9db7d88fae1efb660dce: Status 404 returned error can't find the container with id 22dcb9d758d5e8402e9c6a7981e6c0fe9cd538900d1d9db7d88fae1efb660dce
Jul 27 23:46:55.536878 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:55.536791    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4" event={"ID":"fd3f328b-8c2e-4759-943a-764c8e77f63d","Type":"ContainerStarted","Data":"22dcb9d758d5e8402e9c6a7981e6c0fe9cd538900d1d9db7d88fae1efb660dce"}
Jul 27 23:46:56.542902 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:56.542863    2570 generic.go:356] "Generic (PLEG): container finished" podID="fd3f328b-8c2e-4759-943a-764c8e77f63d" containerID="a801e04eb93b09866c3204afaa97b054a1841ecebdd2d033e84fc7c86dade77f" exitCode=0
Jul 27 23:46:56.543287 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:56.542952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4" event={"ID":"fd3f328b-8c2e-4759-943a-764c8e77f63d","Type":"ContainerDied","Data":"a801e04eb93b09866c3204afaa97b054a1841ecebdd2d033e84fc7c86dade77f"}
Jul 27 23:46:57.683832 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:57.683807    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:57.754427 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:57.754394    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f328b-8c2e-4759-943a-764c8e77f63d-maas-api-service-ca\") pod \"fd3f328b-8c2e-4759-943a-764c8e77f63d\" (UID: \"fd3f328b-8c2e-4759-943a-764c8e77f63d\") "
Jul 27 23:46:57.754740 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:57.754719    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3f328b-8c2e-4759-943a-764c8e77f63d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd3f328b-8c2e-4759-943a-764c8e77f63d" (UID: "fd3f328b-8c2e-4759-943a-764c8e77f63d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:46:57.856087 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:57.856008    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f328b-8c2e-4759-943a-764c8e77f63d-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:46:58.553588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:58.553557    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"
Jul 27 23:46:58.553756 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:58.553587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4" event={"ID":"fd3f328b-8c2e-4759-943a-764c8e77f63d","Type":"ContainerDied","Data":"22dcb9d758d5e8402e9c6a7981e6c0fe9cd538900d1d9db7d88fae1efb660dce"}
Jul 27 23:46:58.553756 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:46:58.553633    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22dcb9d758d5e8402e9c6a7981e6c0fe9cd538900d1d9db7d88fae1efb660dce"
Jul 27 23:47:05.342949 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.342912    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"]
Jul 27 23:47:05.343453 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.343184    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p" podUID="c4fb04a7-210d-42e2-bf70-deed865c4563" containerName="maas-api" containerID="cri-o://7246067ac29c0a377f709a09a0cbb2b993751cc5eb9736f27b48549f75ff99c2" gracePeriod=30
Jul 27 23:47:05.397012 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.396973    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"]
Jul 27 23:47:05.397247 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.397221    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq" podUID="f9744860-4bfa-4f6e-b1b4-e5785b0534cb" containerName="payload-processing" containerID="cri-o://619013fa2de3d38974b0b6cc89e9d1665ac1982ffcb3429e3ac81c8558e975da" gracePeriod=30
Jul 27 23:47:05.403193 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.403162    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"]
Jul 27 23:47:05.403479 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.403443    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6" podUID="ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" containerName="payload-pre-processing" containerID="cri-o://f83bce18744f2d2a03dd55c9af074009d2c36e3413973e9e8243bf73c72743cd" gracePeriod=30
Jul 27 23:47:05.590715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.590640    2570 generic.go:356] "Generic (PLEG): container finished" podID="c4fb04a7-210d-42e2-bf70-deed865c4563" containerID="7246067ac29c0a377f709a09a0cbb2b993751cc5eb9736f27b48549f75ff99c2" exitCode=0
Jul 27 23:47:05.590862 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.590722    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p" event={"ID":"c4fb04a7-210d-42e2-bf70-deed865c4563","Type":"ContainerDied","Data":"7246067ac29c0a377f709a09a0cbb2b993751cc5eb9736f27b48549f75ff99c2"}
Jul 27 23:47:05.593070 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.592989    2570 generic.go:356] "Generic (PLEG): container finished" podID="ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" containerID="f83bce18744f2d2a03dd55c9af074009d2c36e3413973e9e8243bf73c72743cd" exitCode=0
Jul 27 23:47:05.593189 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.593115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6" event={"ID":"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc","Type":"ContainerDied","Data":"f83bce18744f2d2a03dd55c9af074009d2c36e3413973e9e8243bf73c72743cd"}
Jul 27 23:47:05.595088 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.595064    2570 generic.go:356] "Generic (PLEG): container finished" podID="f9744860-4bfa-4f6e-b1b4-e5785b0534cb" containerID="619013fa2de3d38974b0b6cc89e9d1665ac1982ffcb3429e3ac81c8558e975da" exitCode=0
Jul 27 23:47:05.595188 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.595101    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq" event={"ID":"f9744860-4bfa-4f6e-b1b4-e5785b0534cb","Type":"ContainerDied","Data":"619013fa2de3d38974b0b6cc89e9d1665ac1982ffcb3429e3ac81c8558e975da"}
Jul 27 23:47:05.689782 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.689756    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:47:05.718092 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.718070    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:47:05.722111 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.722088    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:47:05.829404 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829342    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv4nl\" (UniqueName: \"kubernetes.io/projected/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-kube-api-access-bv4nl\") pod \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") "
Jul 27 23:47:05.829584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829429    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb04a7-210d-42e2-bf70-deed865c4563-openshift-service-ca\") pod \"c4fb04a7-210d-42e2-bf70-deed865c4563\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") "
Jul 27 23:47:05.829584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829467    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbqzv\" (UniqueName: \"kubernetes.io/projected/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-kube-api-access-mbqzv\") pod \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") "
Jul 27 23:47:05.829584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829490    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xxgc\" (UniqueName: \"kubernetes.io/projected/c4fb04a7-210d-42e2-bf70-deed865c4563-kube-api-access-5xxgc\") pod \"c4fb04a7-210d-42e2-bf70-deed865c4563\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") "
Jul 27 23:47:05.829584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829520    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fb04a7-210d-42e2-bf70-deed865c4563-maas-api-tls\") pod \"c4fb04a7-210d-42e2-bf70-deed865c4563\" (UID: \"c4fb04a7-210d-42e2-bf70-deed865c4563\") "
Jul 27 23:47:05.829783 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829654    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-plugins-config-volume\") pod \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\" (UID: \"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc\") "
Jul 27 23:47:05.829783 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829769    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-plugins-config-volume\") pod \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\" (UID: \"f9744860-4bfa-4f6e-b1b4-e5785b0534cb\") "
Jul 27 23:47:05.829954 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.829893    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fb04a7-210d-42e2-bf70-deed865c4563-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4fb04a7-210d-42e2-bf70-deed865c4563" (UID: "c4fb04a7-210d-42e2-bf70-deed865c4563"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:05.830106 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.830085    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fb04a7-210d-42e2-bf70-deed865c4563-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:05.830106 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.830083    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" (UID: "ca52a04f-a999-44b8-a3ff-d0fd4dee2edc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:05.830240 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.830195    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9744860-4bfa-4f6e-b1b4-e5785b0534cb" (UID: "f9744860-4bfa-4f6e-b1b4-e5785b0534cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:05.832592 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.832564    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-kube-api-access-bv4nl" (OuterVolumeSpecName: "kube-api-access-bv4nl") pod "f9744860-4bfa-4f6e-b1b4-e5785b0534cb" (UID: "f9744860-4bfa-4f6e-b1b4-e5785b0534cb"). InnerVolumeSpecName "kube-api-access-bv4nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:05.832685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.832592    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4fb04a7-210d-42e2-bf70-deed865c4563-kube-api-access-5xxgc" (OuterVolumeSpecName: "kube-api-access-5xxgc") pod "c4fb04a7-210d-42e2-bf70-deed865c4563" (UID: "c4fb04a7-210d-42e2-bf70-deed865c4563"). InnerVolumeSpecName "kube-api-access-5xxgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:05.832685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.832597    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4fb04a7-210d-42e2-bf70-deed865c4563-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4fb04a7-210d-42e2-bf70-deed865c4563" (UID: "c4fb04a7-210d-42e2-bf70-deed865c4563"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:47:05.832685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.832627    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-kube-api-access-mbqzv" (OuterVolumeSpecName: "kube-api-access-mbqzv") pod "ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" (UID: "ca52a04f-a999-44b8-a3ff-d0fd4dee2edc"). InnerVolumeSpecName "kube-api-access-mbqzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:05.931505 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.931473    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4fb04a7-210d-42e2-bf70-deed865c4563-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:05.931505 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.931500    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:05.931706 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.931515    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:05.931706 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.931528    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv4nl\" (UniqueName: \"kubernetes.io/projected/f9744860-4bfa-4f6e-b1b4-e5785b0534cb-kube-api-access-bv4nl\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:05.931706 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.931540    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbqzv\" (UniqueName: \"kubernetes.io/projected/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc-kube-api-access-mbqzv\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:05.931706 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:05.931551    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxgc\" (UniqueName: \"kubernetes.io/projected/c4fb04a7-210d-42e2-bf70-deed865c4563-kube-api-access-5xxgc\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:06.601973 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.601943    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"
Jul 27 23:47:06.601973 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.601962    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6" event={"ID":"ca52a04f-a999-44b8-a3ff-d0fd4dee2edc","Type":"ContainerDied","Data":"406d2fb16975cdbe28618cf74b9324fda546b1b047168d597f6b9291c091b1b6"}
Jul 27 23:47:06.602491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.602029    2570 scope.go:117] "RemoveContainer" containerID="f83bce18744f2d2a03dd55c9af074009d2c36e3413973e9e8243bf73c72743cd"
Jul 27 23:47:06.603440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.603405    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"
Jul 27 23:47:06.603440 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.603424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq" event={"ID":"f9744860-4bfa-4f6e-b1b4-e5785b0534cb","Type":"ContainerDied","Data":"26ee61f6cdc7c0d6271d83e55ffc696ce09d0f8e87dea9883ea7add595514227"}
Jul 27 23:47:06.604895 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.604871    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p" event={"ID":"c4fb04a7-210d-42e2-bf70-deed865c4563","Type":"ContainerDied","Data":"56fa1357e2042fc36d454798c50bfdb6c3488b55a40ccbffdfb9005a9bdb9a41"}
Jul 27 23:47:06.605027 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.604922    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"
Jul 27 23:47:06.612672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.612654    2570 scope.go:117] "RemoveContainer" containerID="619013fa2de3d38974b0b6cc89e9d1665ac1982ffcb3429e3ac81c8558e975da"
Jul 27 23:47:06.621269 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.621253    2570 scope.go:117] "RemoveContainer" containerID="7246067ac29c0a377f709a09a0cbb2b993751cc5eb9736f27b48549f75ff99c2"
Jul 27 23:47:06.628413 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.628388    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"]
Jul 27 23:47:06.633882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.633861    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17ce21b6-676797c4d8-msmx6"]
Jul 27 23:47:06.644952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.644925    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"]
Jul 27 23:47:06.646610 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.646591    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-17ce21b6-75cb997db-kkkrq"]
Jul 27 23:47:06.657693 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.657643    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"]
Jul 27 23:47:06.659344 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:06.659327    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-17ce21b6-5d4cd566ff-llr9p"]
Jul 27 23:47:07.522153 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:07.522118    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4fb04a7-210d-42e2-bf70-deed865c4563" path="/var/lib/kubelet/pods/c4fb04a7-210d-42e2-bf70-deed865c4563/volumes"
Jul 27 23:47:07.522581 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:07.522561    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" path="/var/lib/kubelet/pods/ca52a04f-a999-44b8-a3ff-d0fd4dee2edc/volumes"
Jul 27 23:47:07.522938 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:07.522920    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9744860-4bfa-4f6e-b1b4-e5785b0534cb" path="/var/lib/kubelet/pods/f9744860-4bfa-4f6e-b1b4-e5785b0534cb/volumes"
Jul 27 23:47:09.988161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:09.988126    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"]
Jul 27 23:47:09.989685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:09.989664    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17ce21b6-fsfx4"]
Jul 27 23:47:11.136129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136089    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"]
Jul 27 23:47:11.136577 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136561    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" containerName="payload-pre-processing"
Jul 27 23:47:11.136624 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136579    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" containerName="payload-pre-processing"
Jul 27 23:47:11.136624 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136597    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4fb04a7-210d-42e2-bf70-deed865c4563" containerName="maas-api"
Jul 27 23:47:11.136624 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136602    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4fb04a7-210d-42e2-bf70-deed865c4563" containerName="maas-api"
Jul 27 23:47:11.136624 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136609    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9744860-4bfa-4f6e-b1b4-e5785b0534cb" containerName="payload-processing"
Jul 27 23:47:11.136624 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136614    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9744860-4bfa-4f6e-b1b4-e5785b0534cb" containerName="payload-processing"
Jul 27 23:47:11.136624 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136623    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3f328b-8c2e-4759-943a-764c8e77f63d" containerName="revoke-keys"
Jul 27 23:47:11.136808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136629    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3f328b-8c2e-4759-943a-764c8e77f63d" containerName="revoke-keys"
Jul 27 23:47:11.136808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136684    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9744860-4bfa-4f6e-b1b4-e5785b0534cb" containerName="payload-processing"
Jul 27 23:47:11.136808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136693    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca52a04f-a999-44b8-a3ff-d0fd4dee2edc" containerName="payload-pre-processing"
Jul 27 23:47:11.136808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136699    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3f328b-8c2e-4759-943a-764c8e77f63d" containerName="revoke-keys"
Jul 27 23:47:11.136808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.136707    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4fb04a7-210d-42e2-bf70-deed865c4563" containerName="maas-api"
Jul 27 23:47:11.141616 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.141596    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.144556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.144536    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5c2a4b0b-gw-openshift-default-dockercfg-mwj2q\""
Jul 27 23:47:11.151281 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.151258    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"]
Jul 27 23:47:11.282784 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.282743    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-data\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.282950 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.282799    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-istio-token\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.282950 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.282878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-credential-socket\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.282950 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.282904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1392df87-900c-4572-ac88-07cfcf017e0f-istio-podinfo\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.283066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.282950    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-socket\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.283066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.282998    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhfwt\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-kube-api-access-xhfwt\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.283130 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.283070    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1392df87-900c-4572-ac88-07cfcf017e0f-istiod-ca-cert\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.283130 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.283117    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-envoy\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.283194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.283144    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-certs\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384084 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-envoy\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-certs\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384137    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-data\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-istio-token\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384213    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-credential-socket\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1392df87-900c-4572-ac88-07cfcf017e0f-istio-podinfo\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384456    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-socket\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384484    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhfwt\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-kube-api-access-xhfwt\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384515    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1392df87-900c-4572-ac88-07cfcf017e0f-istiod-ca-cert\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384594    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-certs\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-data\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384831 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384810    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-credential-socket\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.384902 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.384868    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-socket\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.385295 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.385278    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1392df87-900c-4572-ac88-07cfcf017e0f-istiod-ca-cert\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.386528 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.386483    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-envoy\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.387071 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.387048    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1392df87-900c-4572-ac88-07cfcf017e0f-istio-podinfo\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.392338 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.392318    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhfwt\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-kube-api-access-xhfwt\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.392625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.392604    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-istio-token\") pod \"e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") " pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.452544 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.452512    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:11.525580 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.525548    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3f328b-8c2e-4759-943a-764c8e77f63d" path="/var/lib/kubelet/pods/fd3f328b-8c2e-4759-943a-764c8e77f63d/volumes"
Jul 27 23:47:11.596237 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.596210    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"]
Jul 27 23:47:11.597561 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:47:11.597534    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1392df87_900c_4572_ac88_07cfcf017e0f.slice/crio-23c51b135fc763e6e2fe9c47d441a612fa8b44b7acd6d7ad99425f0d18ad00b7 WatchSource:0}: Error finding container 23c51b135fc763e6e2fe9c47d441a612fa8b44b7acd6d7ad99425f0d18ad00b7: Status 404 returned error can't find the container with id 23c51b135fc763e6e2fe9c47d441a612fa8b44b7acd6d7ad99425f0d18ad00b7
Jul 27 23:47:11.633464 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:11.633434    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" event={"ID":"1392df87-900c-4572-ac88-07cfcf017e0f","Type":"ContainerStarted","Data":"23c51b135fc763e6e2fe9c47d441a612fa8b44b7acd6d7ad99425f0d18ad00b7"}
Jul 27 23:47:14.210007 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:14.209966    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:47:14.210281 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:14.210050    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:47:14.210281 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:14.210086    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:47:14.651299 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:14.651258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" event={"ID":"1392df87-900c-4572-ac88-07cfcf017e0f","Type":"ContainerStarted","Data":"db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"}
Jul 27 23:47:14.672666 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:14.672614    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" podStartSLOduration=1.062582689 podStartE2EDuration="3.672598619s" totalImagesPullingTime="2.61001593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:14.670732243 +0000 UTC m=+1859.622075629" watchObservedRunningTime="2026-07-27 23:47:14.672598619 +0000 UTC m=+1859.623942005"
Jul 27 23:47:15.453140 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:15.453100    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:15.457864 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:15.457840    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:15.656084 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:15.656047    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:15.656996 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:15.656975    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:16.188077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.188046    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"]
Jul 27 23:47:16.191612 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.191593    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.194447 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.194427    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5c2a4b0b-serving-cert\""
Jul 27 23:47:16.201152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.201125    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"]
Jul 27 23:47:16.333894 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.333858    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4f18d5-b260-4bad-9fd8-42b49857c50b-openshift-service-ca\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.334049 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.333993    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmlv\" (UniqueName: \"kubernetes.io/projected/8a4f18d5-b260-4bad-9fd8-42b49857c50b-kube-api-access-hpmlv\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.334049 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.334028    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.435118 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.435083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmlv\" (UniqueName: \"kubernetes.io/projected/8a4f18d5-b260-4bad-9fd8-42b49857c50b-kube-api-access-hpmlv\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.435118 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.435125    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.435411 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.435158    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4f18d5-b260-4bad-9fd8-42b49857c50b-openshift-service-ca\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.435411 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:47:16.435248    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-serving-cert: secret "maas-api-e2e-migrate-5c2a4b0b-serving-cert" not found
Jul 27 23:47:16.435411 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:47:16.435328    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls podName:8a4f18d5-b260-4bad-9fd8-42b49857c50b nodeName:}" failed. No retries permitted until 2026-07-27 23:47:16.935308722 +0000 UTC m=+1861.886652087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls") pod "maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" (UID: "8a4f18d5-b260-4bad-9fd8-42b49857c50b") : secret "maas-api-e2e-migrate-5c2a4b0b-serving-cert" not found
Jul 27 23:47:16.435873 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.435856    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4f18d5-b260-4bad-9fd8-42b49857c50b-openshift-service-ca\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.448855 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.448791    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmlv\" (UniqueName: \"kubernetes.io/projected/8a4f18d5-b260-4bad-9fd8-42b49857c50b-kube-api-access-hpmlv\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.878333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.878254    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"]
Jul 27 23:47:16.881846 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.881829    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:16.884739 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.884715    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5c2a4b0b-dockercfg-slb56\""
Jul 27 23:47:16.884856 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.884787    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5c2a4b0b\""
Jul 27 23:47:16.889979 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.889925    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"]
Jul 27 23:47:16.940104 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.940070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:16.942607 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:16.942574    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls\") pod \"maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:17.011198 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.011160    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"]
Jul 27 23:47:17.014680 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.014664    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.024743 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.024720    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"]
Jul 27 23:47:17.041177 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.041134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfbb85c-677b-495d-9b7a-8b69aef67aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.041336 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.041181    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4p9l\" (UniqueName: \"kubernetes.io/projected/4bfbb85c-677b-495d-9b7a-8b69aef67aec-kube-api-access-f4p9l\") pod \"payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.104777 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.104742    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:17.142708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.142667    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfbb85c-677b-495d-9b7a-8b69aef67aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.142708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.142703    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4p9l\" (UniqueName: \"kubernetes.io/projected/4bfbb85c-677b-495d-9b7a-8b69aef67aec-kube-api-access-f4p9l\") pod \"payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.142958 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.142736    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mjf\" (UniqueName: \"kubernetes.io/projected/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-kube-api-access-b4mjf\") pod \"payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") " pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.142958 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.142790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") " pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.143437 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.143416    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfbb85c-677b-495d-9b7a-8b69aef67aec-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.151107 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.151070    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4p9l\" (UniqueName: \"kubernetes.io/projected/4bfbb85c-677b-495d-9b7a-8b69aef67aec-kube-api-access-f4p9l\") pod \"payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.194707 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.194679    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.243595 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.243556    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") " pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.243764 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.243721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mjf\" (UniqueName: \"kubernetes.io/projected/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-kube-api-access-b4mjf\") pod \"payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") " pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.244693 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.244579    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") " pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.247090 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.247071    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"]
Jul 27 23:47:17.249520 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:47:17.249494    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a4f18d5_b260_4bad_9fd8_42b49857c50b.slice/crio-44f6f4acfb9ca13e163ff3fe8cf28031df9db9d3b9b5c22d76b29a64fd445980 WatchSource:0}: Error finding container 44f6f4acfb9ca13e163ff3fe8cf28031df9db9d3b9b5c22d76b29a64fd445980: Status 404 returned error can't find the container with id 44f6f4acfb9ca13e163ff3fe8cf28031df9db9d3b9b5c22d76b29a64fd445980
Jul 27 23:47:17.253799 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.253769    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mjf\" (UniqueName: \"kubernetes.io/projected/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-kube-api-access-b4mjf\") pod \"payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") " pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.327813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.327782    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.469023 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.468992    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"]
Jul 27 23:47:17.470141 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:47:17.470114    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb2155a_2d57_475a_9ff9_5ea52c0ffd25.slice/crio-e2ff9074ebaab214515704f6a7a0a66b0b1863d348323cb294e0eddf3ea75faa WatchSource:0}: Error finding container e2ff9074ebaab214515704f6a7a0a66b0b1863d348323cb294e0eddf3ea75faa: Status 404 returned error can't find the container with id e2ff9074ebaab214515704f6a7a0a66b0b1863d348323cb294e0eddf3ea75faa
Jul 27 23:47:17.545630 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.545603    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"]
Jul 27 23:47:17.546489 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:47:17.546461    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfbb85c_677b_495d_9b7a_8b69aef67aec.slice/crio-3d741b91bd51ad5d06b005c6b5274e0ffc057a53494ce7072a29ec1f83a49642 WatchSource:0}: Error finding container 3d741b91bd51ad5d06b005c6b5274e0ffc057a53494ce7072a29ec1f83a49642: Status 404 returned error can't find the container with id 3d741b91bd51ad5d06b005c6b5274e0ffc057a53494ce7072a29ec1f83a49642
Jul 27 23:47:17.668141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.668075    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" event={"ID":"8a4f18d5-b260-4bad-9fd8-42b49857c50b","Type":"ContainerStarted","Data":"44f6f4acfb9ca13e163ff3fe8cf28031df9db9d3b9b5c22d76b29a64fd445980"}
Jul 27 23:47:17.670117 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.670088    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966" event={"ID":"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25","Type":"ContainerStarted","Data":"7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03"}
Jul 27 23:47:17.670238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.670141    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966" event={"ID":"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25","Type":"ContainerStarted","Data":"e2ff9074ebaab214515704f6a7a0a66b0b1863d348323cb294e0eddf3ea75faa"}
Jul 27 23:47:17.670238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.670171    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:17.671695 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.671672    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n" event={"ID":"4bfbb85c-677b-495d-9b7a-8b69aef67aec","Type":"ContainerStarted","Data":"dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718"}
Jul 27 23:47:17.671818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.671719    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n" event={"ID":"4bfbb85c-677b-495d-9b7a-8b69aef67aec","Type":"ContainerStarted","Data":"3d741b91bd51ad5d06b005c6b5274e0ffc057a53494ce7072a29ec1f83a49642"}
Jul 27 23:47:17.671883 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.671832    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:17.688113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.688034    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966" podStartSLOduration=1.688016271 podStartE2EDuration="1.688016271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:17.687417611 +0000 UTC m=+1862.638761019" watchObservedRunningTime="2026-07-27 23:47:17.688016271 +0000 UTC m=+1862.639359659"
Jul 27 23:47:17.703997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:17.703958    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n" podStartSLOduration=1.703944436 podStartE2EDuration="1.703944436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:17.701884188 +0000 UTC m=+1862.653227595" watchObservedRunningTime="2026-07-27 23:47:17.703944436 +0000 UTC m=+1862.655287823"
Jul 27 23:47:18.684309 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:18.684266    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" event={"ID":"8a4f18d5-b260-4bad-9fd8-42b49857c50b","Type":"ContainerStarted","Data":"b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd"}
Jul 27 23:47:18.702173 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:18.702097    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" podStartSLOduration=2.305217789 podStartE2EDuration="2.702084013s" totalImagesPullingTime="396.866224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:47:18.70190096 +0000 UTC m=+1863.653244347" watchObservedRunningTime="2026-07-27 23:47:18.702084013 +0000 UTC m=+1863.653427401"
Jul 27 23:47:19.689185 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:19.689152    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:25.699820 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:25.699788    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:27.732201 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:27.732162    2570 generic.go:356] "Generic (PLEG): container finished" podID="1392df87-900c-4572-ac88-07cfcf017e0f" containerID="db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69" exitCode=0
Jul 27 23:47:27.732633 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:27.732227    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" event={"ID":"1392df87-900c-4572-ac88-07cfcf017e0f","Type":"ContainerDied","Data":"db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"}
Jul 27 23:47:27.732633 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:27.732617    2570 scope.go:117] "RemoveContainer" containerID="db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"
Jul 27 23:47:27.733631 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:27.733591    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:47:27.733745 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:27.733684    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:47:27.733745 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:27.733736    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:47:28.686452 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:28.686424    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:28.686641 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:28.686501    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:28.740596 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:28.740563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" event={"ID":"1392df87-900c-4572-ac88-07cfcf017e0f","Type":"ContainerStarted","Data":"ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937"}
Jul 27 23:47:29.453337 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:29.453301    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:29.454794 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:29.454766    2570 patch_prober.go:28] interesting pod/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 23:47:29.455042 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:29.455001    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 23:47:30.453846 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:30.453808    2570 patch_prober.go:28] interesting pod/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 23:47:30.454494 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:30.454458    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 23:47:30.656652 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:30.656610    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:31.453132 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:31.453088    2570 patch_prober.go:28] interesting pod/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 23:47:31.453376 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:31.453154    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 23:47:32.453973 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:32.453924    2570 patch_prober.go:28] interesting pod/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 23:47:32.454498 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:32.454020    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 23:47:33.457956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:33.457926    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:33.459123 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:33.459106    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:47:51.477409 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.477345    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"]
Jul 27 23:47:51.477998 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.477626    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" podUID="8a4f18d5-b260-4bad-9fd8-42b49857c50b" containerName="maas-api" containerID="cri-o://b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd" gracePeriod=30
Jul 27 23:47:51.526518 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.526433    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"]
Jul 27 23:47:51.526733 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.526664    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966" podUID="dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" containerName="payload-processing" containerID="cri-o://7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03" gracePeriod=30
Jul 27 23:47:51.531487 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.531291    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"]
Jul 27 23:47:51.531737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.531536    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n" podUID="4bfbb85c-677b-495d-9b7a-8b69aef67aec" containerName="payload-pre-processing" containerID="cri-o://dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718" gracePeriod=30
Jul 27 23:47:51.797196 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.797160    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:51.825786 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.825760    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:51.830206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.830186    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:51.860766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.860728    2570 generic.go:356] "Generic (PLEG): container finished" podID="dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" containerID="7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03" exitCode=0
Jul 27 23:47:51.860766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.860767    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"
Jul 27 23:47:51.860766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.860768    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966" event={"ID":"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25","Type":"ContainerDied","Data":"7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03"}
Jul 27 23:47:51.861051 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.860824    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966" event={"ID":"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25","Type":"ContainerDied","Data":"e2ff9074ebaab214515704f6a7a0a66b0b1863d348323cb294e0eddf3ea75faa"}
Jul 27 23:47:51.861051 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.860857    2570 scope.go:117] "RemoveContainer" containerID="7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03"
Jul 27 23:47:51.862254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.862229    2570 generic.go:356] "Generic (PLEG): container finished" podID="4bfbb85c-677b-495d-9b7a-8b69aef67aec" containerID="dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718" exitCode=0
Jul 27 23:47:51.862396 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.862271    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"
Jul 27 23:47:51.862396 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.862299    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n" event={"ID":"4bfbb85c-677b-495d-9b7a-8b69aef67aec","Type":"ContainerDied","Data":"dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718"}
Jul 27 23:47:51.862396 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.862345    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n" event={"ID":"4bfbb85c-677b-495d-9b7a-8b69aef67aec","Type":"ContainerDied","Data":"3d741b91bd51ad5d06b005c6b5274e0ffc057a53494ce7072a29ec1f83a49642"}
Jul 27 23:47:51.868441 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.868412    2570 generic.go:356] "Generic (PLEG): container finished" podID="8a4f18d5-b260-4bad-9fd8-42b49857c50b" containerID="b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd" exitCode=0
Jul 27 23:47:51.868597 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.868493    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"
Jul 27 23:47:51.868597 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.868500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" event={"ID":"8a4f18d5-b260-4bad-9fd8-42b49857c50b","Type":"ContainerDied","Data":"b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd"}
Jul 27 23:47:51.868597 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.868555    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw" event={"ID":"8a4f18d5-b260-4bad-9fd8-42b49857c50b","Type":"ContainerDied","Data":"44f6f4acfb9ca13e163ff3fe8cf28031df9db9d3b9b5c22d76b29a64fd445980"}
Jul 27 23:47:51.872857 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.872837    2570 scope.go:117] "RemoveContainer" containerID="7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03"
Jul 27 23:47:51.873123 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:47:51.873104    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03\": container with ID starting with 7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03 not found: ID does not exist" containerID="7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03"
Jul 27 23:47:51.873205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873137    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03"} err="failed to get container status \"7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03\": rpc error: code = NotFound desc = could not find container \"7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03\": container with ID starting with 7288072d049060d9394135089f88be362549ca3971ba1f943b887fbb0e4fdc03 not found: ID does not exist"
Jul 27 23:47:51.873205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873161    2570 scope.go:117] "RemoveContainer" containerID="dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718"
Jul 27 23:47:51.873205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873199    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4f18d5-b260-4bad-9fd8-42b49857c50b-openshift-service-ca\") pod \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") "
Jul 27 23:47:51.873410 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873279    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpmlv\" (UniqueName: \"kubernetes.io/projected/8a4f18d5-b260-4bad-9fd8-42b49857c50b-kube-api-access-hpmlv\") pod \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") "
Jul 27 23:47:51.873410 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls\") pod \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\" (UID: \"8a4f18d5-b260-4bad-9fd8-42b49857c50b\") "
Jul 27 23:47:51.873567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873549    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4f18d5-b260-4bad-9fd8-42b49857c50b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a4f18d5-b260-4bad-9fd8-42b49857c50b" (UID: "8a4f18d5-b260-4bad-9fd8-42b49857c50b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:51.873643 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.873632    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a4f18d5-b260-4bad-9fd8-42b49857c50b-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:51.876067 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.876040    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a4f18d5-b260-4bad-9fd8-42b49857c50b" (UID: "8a4f18d5-b260-4bad-9fd8-42b49857c50b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:47:51.876067 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.876048    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4f18d5-b260-4bad-9fd8-42b49857c50b-kube-api-access-hpmlv" (OuterVolumeSpecName: "kube-api-access-hpmlv") pod "8a4f18d5-b260-4bad-9fd8-42b49857c50b" (UID: "8a4f18d5-b260-4bad-9fd8-42b49857c50b"). InnerVolumeSpecName "kube-api-access-hpmlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:51.882814 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.882796    2570 scope.go:117] "RemoveContainer" containerID="dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718"
Jul 27 23:47:51.883078 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:47:51.883060    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718\": container with ID starting with dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718 not found: ID does not exist" containerID="dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718"
Jul 27 23:47:51.883129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.883083    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718"} err="failed to get container status \"dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718\": rpc error: code = NotFound desc = could not find container \"dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718\": container with ID starting with dec0a01fafb96505e9cb9e500dedb404b8ffa58fc4952e0c5efb7208d2255718 not found: ID does not exist"
Jul 27 23:47:51.883129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.883099    2570 scope.go:117] "RemoveContainer" containerID="b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd"
Jul 27 23:47:51.892376 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.892336    2570 scope.go:117] "RemoveContainer" containerID="b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd"
Jul 27 23:47:51.892648 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:47:51.892622    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd\": container with ID starting with b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd not found: ID does not exist" containerID="b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd"
Jul 27 23:47:51.892700 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.892646    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd"} err="failed to get container status \"b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd\": rpc error: code = NotFound desc = could not find container \"b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd\": container with ID starting with b20111b8a5239f63609e6c52926c73259f9ab36738f021056bf7cd09221387cd not found: ID does not exist"
Jul 27 23:47:51.974181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974144    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-plugins-config-volume\") pod \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") "
Jul 27 23:47:51.974181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974184    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfbb85c-677b-495d-9b7a-8b69aef67aec-plugins-config-volume\") pod \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") "
Jul 27 23:47:51.974446 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974210    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4mjf\" (UniqueName: \"kubernetes.io/projected/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-kube-api-access-b4mjf\") pod \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\" (UID: \"dbb2155a-2d57-475a-9ff9-5ea52c0ffd25\") "
Jul 27 23:47:51.974446 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974278    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4p9l\" (UniqueName: \"kubernetes.io/projected/4bfbb85c-677b-495d-9b7a-8b69aef67aec-kube-api-access-f4p9l\") pod \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\" (UID: \"4bfbb85c-677b-495d-9b7a-8b69aef67aec\") "
Jul 27 23:47:51.974558 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974484    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpmlv\" (UniqueName: \"kubernetes.io/projected/8a4f18d5-b260-4bad-9fd8-42b49857c50b-kube-api-access-hpmlv\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:51.974558 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974498    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a4f18d5-b260-4bad-9fd8-42b49857c50b-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:51.974678 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974652    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" (UID: "dbb2155a-2d57-475a-9ff9-5ea52c0ffd25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:51.974724 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.974651    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfbb85c-677b-495d-9b7a-8b69aef67aec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4bfbb85c-677b-495d-9b7a-8b69aef67aec" (UID: "4bfbb85c-677b-495d-9b7a-8b69aef67aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:47:51.976575 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.976553    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-kube-api-access-b4mjf" (OuterVolumeSpecName: "kube-api-access-b4mjf") pod "dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" (UID: "dbb2155a-2d57-475a-9ff9-5ea52c0ffd25"). InnerVolumeSpecName "kube-api-access-b4mjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:51.976640 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:51.976584    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfbb85c-677b-495d-9b7a-8b69aef67aec-kube-api-access-f4p9l" (OuterVolumeSpecName: "kube-api-access-f4p9l") pod "4bfbb85c-677b-495d-9b7a-8b69aef67aec" (UID: "4bfbb85c-677b-495d-9b7a-8b69aef67aec"). InnerVolumeSpecName "kube-api-access-f4p9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:47:52.076009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.075900    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:52.076009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.075935    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfbb85c-677b-495d-9b7a-8b69aef67aec-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:52.076009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.075949    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mjf\" (UniqueName: \"kubernetes.io/projected/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25-kube-api-access-b4mjf\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:52.076009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.075963    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4p9l\" (UniqueName: \"kubernetes.io/projected/4bfbb85c-677b-495d-9b7a-8b69aef67aec-kube-api-access-f4p9l\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:47:52.186159 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.186129    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"]
Jul 27 23:47:52.187804 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.187777    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5c2a4b0b-7dd886bc45-rj966"]
Jul 27 23:47:52.195975 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.195951    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"]
Jul 27 23:47:52.198308 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.198288    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5c2a4b0b-87bcf9bbd-q9p6n"]
Jul 27 23:47:52.207231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.207212    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"]
Jul 27 23:47:52.209702 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:52.209681    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5c2a4b0b-77744d6996-4j4nw"]
Jul 27 23:47:53.522960 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:53.522923    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bfbb85c-677b-495d-9b7a-8b69aef67aec" path="/var/lib/kubelet/pods/4bfbb85c-677b-495d-9b7a-8b69aef67aec/volumes"
Jul 27 23:47:53.523344 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:53.523312    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a4f18d5-b260-4bad-9fd8-42b49857c50b" path="/var/lib/kubelet/pods/8a4f18d5-b260-4bad-9fd8-42b49857c50b/volumes"
Jul 27 23:47:53.523656 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:53.523642    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" path="/var/lib/kubelet/pods/dbb2155a-2d57-475a-9ff9-5ea52c0ffd25/volumes"
Jul 27 23:47:56.323542 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:56.323505    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"]
Jul 27 23:47:56.324056 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:56.323765    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy" containerID="cri-o://ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937" gracePeriod=30
Jul 27 23:47:59.784833 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.784796    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"]
Jul 27 23:47:59.785394 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785372    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfbb85c-677b-495d-9b7a-8b69aef67aec" containerName="payload-pre-processing"
Jul 27 23:47:59.785462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785397    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfbb85c-677b-495d-9b7a-8b69aef67aec" containerName="payload-pre-processing"
Jul 27 23:47:59.785462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785416    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a4f18d5-b260-4bad-9fd8-42b49857c50b" containerName="maas-api"
Jul 27 23:47:59.785462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785425    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a4f18d5-b260-4bad-9fd8-42b49857c50b" containerName="maas-api"
Jul 27 23:47:59.785573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785460    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" containerName="payload-processing"
Jul 27 23:47:59.785573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785471    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" containerName="payload-processing"
Jul 27 23:47:59.785573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785551    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bfbb85c-677b-495d-9b7a-8b69aef67aec" containerName="payload-pre-processing"
Jul 27 23:47:59.785573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785562    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbb2155a-2d57-475a-9ff9-5ea52c0ffd25" containerName="payload-processing"
Jul 27 23:47:59.785573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.785571    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a4f18d5-b260-4bad-9fd8-42b49857c50b" containerName="maas-api"
Jul 27 23:47:59.790268 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.790250    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:47:59.792800 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.792780    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7355578b-serving-cert\""
Jul 27 23:47:59.801433 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.801409    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"]
Jul 27 23:47:59.944802 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.944771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6z8f\" (UniqueName: \"kubernetes.io/projected/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-kube-api-access-q6z8f\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:47:59.944954 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.944809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-openshift-service-ca\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:47:59.944954 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:47:59.944907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.046217 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.046116    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.046217 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.046209    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6z8f\" (UniqueName: \"kubernetes.io/projected/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-kube-api-access-q6z8f\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.046446 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:48:00.046285    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-serving-cert: secret "maas-api-e2e-ait-7355578b-serving-cert" not found
Jul 27 23:48:00.046446 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:48:00.046380    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls podName:2c2ef2c6-caa1-468a-bfb3-e6bb4464163e nodeName:}" failed. No retries permitted until 2026-07-27 23:48:00.546328771 +0000 UTC m=+1905.497672146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls") pod "maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" (UID: "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e") : secret "maas-api-e2e-ait-7355578b-serving-cert" not found
Jul 27 23:48:00.046446 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.046403    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-openshift-service-ca\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.047113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.047091    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-openshift-service-ca\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.057705 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.057671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6z8f\" (UniqueName: \"kubernetes.io/projected/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-kube-api-access-q6z8f\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.366495 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.366398    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"]
Jul 27 23:48:00.370113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.370079    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.372877 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.372853    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7355578b\""
Jul 27 23:48:00.372984 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.372925    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7355578b-dockercfg-tc6bs\""
Jul 27 23:48:00.380700 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.380671    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"]
Jul 27 23:48:00.397449 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.397413    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"]
Jul 27 23:48:00.400758 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.400741    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.408018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.407952    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"]
Jul 27 23:48:00.551152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.551115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9znf\" (UniqueName: \"kubernetes.io/projected/1974feb8-6d42-4d91-91e4-d20f118627eb-kube-api-access-c9znf\") pod \"payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.551152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.551158    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.551406 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.551178    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5ec78-d523-4257-9971-7fc641045a06-plugins-config-volume\") pod \"payload-processing-e2e-ait-7355578b-85689f45d4-tbbph\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") " pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.551406 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.551271    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzks\" (UniqueName: \"kubernetes.io/projected/a7f5ec78-d523-4257-9971-7fc641045a06-kube-api-access-jdzks\") pod \"payload-processing-e2e-ait-7355578b-85689f45d4-tbbph\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") " pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.551406 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.551307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1974feb8-6d42-4d91-91e4-d20f118627eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.553750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.553732    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls\") pod \"maas-api-e2e-ait-7355578b-5bd8c56568-97fpb\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.652218 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.652182    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9znf\" (UniqueName: \"kubernetes.io/projected/1974feb8-6d42-4d91-91e4-d20f118627eb-kube-api-access-c9znf\") pod \"payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.652218 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.652224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5ec78-d523-4257-9971-7fc641045a06-plugins-config-volume\") pod \"payload-processing-e2e-ait-7355578b-85689f45d4-tbbph\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") " pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.652541 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.652268    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzks\" (UniqueName: \"kubernetes.io/projected/a7f5ec78-d523-4257-9971-7fc641045a06-kube-api-access-jdzks\") pod \"payload-processing-e2e-ait-7355578b-85689f45d4-tbbph\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") " pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.652541 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.652293    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1974feb8-6d42-4d91-91e4-d20f118627eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.653029 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.653008    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1974feb8-6d42-4d91-91e4-d20f118627eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.653029 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.653021    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5ec78-d523-4257-9971-7fc641045a06-plugins-config-volume\") pod \"payload-processing-e2e-ait-7355578b-85689f45d4-tbbph\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") " pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.661380 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.661322    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9znf\" (UniqueName: \"kubernetes.io/projected/1974feb8-6d42-4d91-91e4-d20f118627eb-kube-api-access-c9znf\") pod \"payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.661484 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.661322    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzks\" (UniqueName: \"kubernetes.io/projected/a7f5ec78-d523-4257-9971-7fc641045a06-kube-api-access-jdzks\") pod \"payload-processing-e2e-ait-7355578b-85689f45d4-tbbph\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") " pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.681839 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.681815    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:00.702688 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.702661    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:00.711890 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.711857    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:00.843606 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.843500    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"]
Jul 27 23:48:00.848068 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:00.848020    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1974feb8_6d42_4d91_91e4_d20f118627eb.slice/crio-6e90045b01d282826eb5a3903e5b743caae724d35e5e47b093df29f6d2d247b9 WatchSource:0}: Error finding container 6e90045b01d282826eb5a3903e5b743caae724d35e5e47b093df29f6d2d247b9: Status 404 returned error can't find the container with id 6e90045b01d282826eb5a3903e5b743caae724d35e5e47b093df29f6d2d247b9
Jul 27 23:48:00.876955 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.876877    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"]
Jul 27 23:48:00.886805 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:00.886763    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2ef2c6_caa1_468a_bfb3_e6bb4464163e.slice/crio-f0d5ff21d47f4206792811b6ed963e013548360848185e58e5e74c6c72fc273e WatchSource:0}: Error finding container f0d5ff21d47f4206792811b6ed963e013548360848185e58e5e74c6c72fc273e: Status 404 returned error can't find the container with id f0d5ff21d47f4206792811b6ed963e013548360848185e58e5e74c6c72fc273e
Jul 27 23:48:00.908680 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.908647    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"]
Jul 27 23:48:00.911596 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:00.911571    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f5ec78_d523_4257_9971_7fc641045a06.slice/crio-45cbdd005ea9e7977d499b778808b248bc5cff0402a03ac4a055bdda601536bf WatchSource:0}: Error finding container 45cbdd005ea9e7977d499b778808b248bc5cff0402a03ac4a055bdda601536bf: Status 404 returned error can't find the container with id 45cbdd005ea9e7977d499b778808b248bc5cff0402a03ac4a055bdda601536bf
Jul 27 23:48:00.918855 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.918821    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2" event={"ID":"1974feb8-6d42-4d91-91e4-d20f118627eb","Type":"ContainerStarted","Data":"6e90045b01d282826eb5a3903e5b743caae724d35e5e47b093df29f6d2d247b9"}
Jul 27 23:48:00.920152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.920088    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph" event={"ID":"a7f5ec78-d523-4257-9971-7fc641045a06","Type":"ContainerStarted","Data":"45cbdd005ea9e7977d499b778808b248bc5cff0402a03ac4a055bdda601536bf"}
Jul 27 23:48:00.921236 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:00.921210    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" event={"ID":"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e","Type":"ContainerStarted","Data":"f0d5ff21d47f4206792811b6ed963e013548360848185e58e5e74c6c72fc273e"}
Jul 27 23:48:01.597222 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.597144    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:48:01.762947 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.762912    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-data\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.762947 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.762950    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-socket\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.762984    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-certs\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763034    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-istio-token\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763062    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhfwt\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-kube-api-access-xhfwt\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763090    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-credential-socket\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763180    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1392df87-900c-4572-ac88-07cfcf017e0f-istiod-ca-cert\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763502 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763233    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1392df87-900c-4572-ac88-07cfcf017e0f-istio-podinfo\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763502 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763267    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-envoy\") pod \"1392df87-900c-4572-ac88-07cfcf017e0f\" (UID: \"1392df87-900c-4572-ac88-07cfcf017e0f\") "
Jul 27 23:48:01.763940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763226    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:01.764139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763339    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:01.764445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763412    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:01.764445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763901    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1392df87-900c-4572-ac88-07cfcf017e0f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:48:01.764445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.763999    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:01.765723 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.765661    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:01.765933 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.765904    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1392df87-900c-4572-ac88-07cfcf017e0f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:48:01.766031 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.766016    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:48:01.766100 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.766075    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-kube-api-access-xhfwt" (OuterVolumeSpecName: "kube-api-access-xhfwt") pod "1392df87-900c-4572-ac88-07cfcf017e0f" (UID: "1392df87-900c-4572-ac88-07cfcf017e0f"). InnerVolumeSpecName "kube-api-access-xhfwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864659    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864689    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864700    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864708    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864717    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhfwt\" (UniqueName: \"kubernetes.io/projected/1392df87-900c-4572-ac88-07cfcf017e0f-kube-api-access-xhfwt\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864727    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864735    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1392df87-900c-4572-ac88-07cfcf017e0f-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864743    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1392df87-900c-4572-ac88-07cfcf017e0f-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.864749 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.864750    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1392df87-900c-4572-ac88-07cfcf017e0f-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:01.927802 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.927765    2570 generic.go:356] "Generic (PLEG): container finished" podID="1392df87-900c-4572-ac88-07cfcf017e0f" containerID="ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937" exitCode=0
Jul 27 23:48:01.927983 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.927813    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"
Jul 27 23:48:01.927983 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.927862    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" event={"ID":"1392df87-900c-4572-ac88-07cfcf017e0f","Type":"ContainerDied","Data":"ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937"}
Jul 27 23:48:01.927983 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.927921    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j" event={"ID":"1392df87-900c-4572-ac88-07cfcf017e0f","Type":"ContainerDied","Data":"23c51b135fc763e6e2fe9c47d441a612fa8b44b7acd6d7ad99425f0d18ad00b7"}
Jul 27 23:48:01.927983 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.927953    2570 scope.go:117] "RemoveContainer" containerID="ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937"
Jul 27 23:48:01.929491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.929467    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph" event={"ID":"a7f5ec78-d523-4257-9971-7fc641045a06","Type":"ContainerStarted","Data":"b0f7a0c258e309f25aa5479be3788e8b4a2a58790d30e98bccd3bfd25f67dd99"}
Jul 27 23:48:01.929611 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.929523    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:01.931287 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.931248    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" event={"ID":"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e","Type":"ContainerStarted","Data":"0229262636f50abff4ee4fb986474ddf720ba547242badc83e58b698eed694ed"}
Jul 27 23:48:01.931445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.931314    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:01.932881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.932860    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2" event={"ID":"1974feb8-6d42-4d91-91e4-d20f118627eb","Type":"ContainerStarted","Data":"ad642e363d125208a3458e156206f95f974ed757de207a90e282842cb346cf35"}
Jul 27 23:48:01.932989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.932981    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:01.938085 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.938070    2570 scope.go:117] "RemoveContainer" containerID="db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"
Jul 27 23:48:01.946067 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.946049    2570 scope.go:117] "RemoveContainer" containerID="ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937"
Jul 27 23:48:01.946318 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:48:01.946297    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937\": container with ID starting with ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937 not found: ID does not exist" containerID="ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937"
Jul 27 23:48:01.946411 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.946329    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937"} err="failed to get container status \"ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937\": rpc error: code = NotFound desc = could not find container \"ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937\": container with ID starting with ebe7c9fb169aaae387cbb98117a7186f23a8bf18bab51c01d1f782003477b937 not found: ID does not exist"
Jul 27 23:48:01.946411 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.946370    2570 scope.go:117] "RemoveContainer" containerID="db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"
Jul 27 23:48:01.946644 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:48:01.946625    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69\": container with ID starting with db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69 not found: ID does not exist" containerID="db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"
Jul 27 23:48:01.946698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.946653    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69"} err="failed to get container status \"db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69\": rpc error: code = NotFound desc = could not find container \"db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69\": container with ID starting with db3ce6fff4741ab1368372fff22c9df84c42f205c59c9c3c983af7ba70c51a69 not found: ID does not exist"
Jul 27 23:48:01.948240 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.948199    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph" podStartSLOduration=1.948184753 podStartE2EDuration="1.948184753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:01.947471196 +0000 UTC m=+1906.898814616" watchObservedRunningTime="2026-07-27 23:48:01.948184753 +0000 UTC m=+1906.899528141"
Jul 27 23:48:01.963532 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.963488    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" podStartSLOduration=2.506405956 podStartE2EDuration="2.963471582s" totalImagesPullingTime="457.065626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:01.962453464 +0000 UTC m=+1906.913796852" watchObservedRunningTime="2026-07-27 23:48:01.963471582 +0000 UTC m=+1906.914814970"
Jul 27 23:48:01.978721 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.978670    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2" podStartSLOduration=1.978656951 podStartE2EDuration="1.978656951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:01.976331385 +0000 UTC m=+1906.927674792" watchObservedRunningTime="2026-07-27 23:48:01.978656951 +0000 UTC m=+1906.930000386"
Jul 27 23:48:01.989237 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.989211    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"]
Jul 27 23:48:01.994704 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:01.994681    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5c2a4b0b-gw-openshift-default-589b95bb44-5jn5j"]
Jul 27 23:48:03.522513 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:03.522483    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" path="/var/lib/kubelet/pods/1392df87-900c-4572-ac88-07cfcf017e0f/volumes"
Jul 27 23:48:07.944699 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:07.944669    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:12.941438 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:12.941406    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:12.941889 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:12.941494    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:25.480481 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.480443    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"]
Jul 27 23:48:25.480934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.480812    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy"
Jul 27 23:48:25.480934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.480822    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy"
Jul 27 23:48:25.480934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.480895    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy"
Jul 27 23:48:25.480934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.480904    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy"
Jul 27 23:48:25.484228 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.484213    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:25.486863 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.486834    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 27 23:48:25.490998 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.490969    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"]
Jul 27 23:48:25.579095 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.579047    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55249e52-3c15-4570-9583-c0574cde6ff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7355578b-wfcqm\" (UID: \"55249e52-3c15-4570-9583-c0574cde6ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:25.679731 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.679691    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55249e52-3c15-4570-9583-c0574cde6ff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7355578b-wfcqm\" (UID: \"55249e52-3c15-4570-9583-c0574cde6ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:25.680288 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.680271    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55249e52-3c15-4570-9583-c0574cde6ff2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7355578b-wfcqm\" (UID: \"55249e52-3c15-4570-9583-c0574cde6ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:25.798319 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.798225    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:25.948892 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:25.948868    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"]
Jul 27 23:48:25.952139 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:25.952108    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55249e52_3c15_4570_9583_c0574cde6ff2.slice/crio-99e5197cfc818217d90852f11065bc6a0d4bb9ba3cddea2603afb037fbfafdf8 WatchSource:0}: Error finding container 99e5197cfc818217d90852f11065bc6a0d4bb9ba3cddea2603afb037fbfafdf8: Status 404 returned error can't find the container with id 99e5197cfc818217d90852f11065bc6a0d4bb9ba3cddea2603afb037fbfafdf8
Jul 27 23:48:26.052083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:26.051999    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm" event={"ID":"55249e52-3c15-4570-9583-c0574cde6ff2","Type":"ContainerStarted","Data":"99e5197cfc818217d90852f11065bc6a0d4bb9ba3cddea2603afb037fbfafdf8"}
Jul 27 23:48:27.059122 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:27.059016    2570 generic.go:356] "Generic (PLEG): container finished" podID="55249e52-3c15-4570-9583-c0574cde6ff2" containerID="cd102a3bca103e5ea1376481d4733344f653d8b2e531b25f77dd620c59385167" exitCode=0
Jul 27 23:48:27.059122 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:27.059107    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm" event={"ID":"55249e52-3c15-4570-9583-c0574cde6ff2","Type":"ContainerDied","Data":"cd102a3bca103e5ea1376481d4733344f653d8b2e531b25f77dd620c59385167"}
Jul 27 23:48:28.216741 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:28.216714    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:28.305485 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:28.305453    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55249e52-3c15-4570-9583-c0574cde6ff2-maas-api-service-ca\") pod \"55249e52-3c15-4570-9583-c0574cde6ff2\" (UID: \"55249e52-3c15-4570-9583-c0574cde6ff2\") "
Jul 27 23:48:28.305813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:28.305789    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55249e52-3c15-4570-9583-c0574cde6ff2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55249e52-3c15-4570-9583-c0574cde6ff2" (UID: "55249e52-3c15-4570-9583-c0574cde6ff2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:48:28.406987 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:28.406954    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55249e52-3c15-4570-9583-c0574cde6ff2-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:29.070124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:29.070089    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"
Jul 27 23:48:29.070299 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:29.070126    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm" event={"ID":"55249e52-3c15-4570-9583-c0574cde6ff2","Type":"ContainerDied","Data":"99e5197cfc818217d90852f11065bc6a0d4bb9ba3cddea2603afb037fbfafdf8"}
Jul 27 23:48:29.070299 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:29.070174    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e5197cfc818217d90852f11065bc6a0d4bb9ba3cddea2603afb037fbfafdf8"
Jul 27 23:48:35.819995 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:35.819961    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"]
Jul 27 23:48:35.820457 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:35.820167    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" podUID="2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" containerName="maas-api" containerID="cri-o://0229262636f50abff4ee4fb986474ddf720ba547242badc83e58b698eed694ed" gracePeriod=30
Jul 27 23:48:35.868104 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:35.868067    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"]
Jul 27 23:48:35.868367 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:35.868311    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph" podUID="a7f5ec78-d523-4257-9971-7fc641045a06" containerName="payload-processing" containerID="cri-o://b0f7a0c258e309f25aa5479be3788e8b4a2a58790d30e98bccd3bfd25f67dd99" gracePeriod=30
Jul 27 23:48:35.878210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:35.878177    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"]
Jul 27 23:48:35.878468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:35.878434    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2" podUID="1974feb8-6d42-4d91-91e4-d20f118627eb" containerName="payload-pre-processing" containerID="cri-o://ad642e363d125208a3458e156206f95f974ed757de207a90e282842cb346cf35" gracePeriod=30
Jul 27 23:48:36.104582 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.104440    2570 generic.go:356] "Generic (PLEG): container finished" podID="a7f5ec78-d523-4257-9971-7fc641045a06" containerID="b0f7a0c258e309f25aa5479be3788e8b4a2a58790d30e98bccd3bfd25f67dd99" exitCode=0
Jul 27 23:48:36.104582 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.104519    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph" event={"ID":"a7f5ec78-d523-4257-9971-7fc641045a06","Type":"ContainerDied","Data":"b0f7a0c258e309f25aa5479be3788e8b4a2a58790d30e98bccd3bfd25f67dd99"}
Jul 27 23:48:36.106431 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.106403    2570 generic.go:356] "Generic (PLEG): container finished" podID="2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" containerID="0229262636f50abff4ee4fb986474ddf720ba547242badc83e58b698eed694ed" exitCode=0
Jul 27 23:48:36.106581 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.106559    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" event={"ID":"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e","Type":"ContainerDied","Data":"0229262636f50abff4ee4fb986474ddf720ba547242badc83e58b698eed694ed"}
Jul 27 23:48:36.108565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.108544    2570 generic.go:356] "Generic (PLEG): container finished" podID="1974feb8-6d42-4d91-91e4-d20f118627eb" containerID="ad642e363d125208a3458e156206f95f974ed757de207a90e282842cb346cf35" exitCode=0
Jul 27 23:48:36.108665 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.108590    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2" event={"ID":"1974feb8-6d42-4d91-91e4-d20f118627eb","Type":"ContainerDied","Data":"ad642e363d125208a3458e156206f95f974ed757de207a90e282842cb346cf35"}
Jul 27 23:48:36.145280 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.145252    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:36.148423 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.148400    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:36.182506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.182481    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:36.286574 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286543    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6z8f\" (UniqueName: \"kubernetes.io/projected/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-kube-api-access-q6z8f\") pod \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") "
Jul 27 23:48:36.286744 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286639    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls\") pod \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") "
Jul 27 23:48:36.286744 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286711    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5ec78-d523-4257-9971-7fc641045a06-plugins-config-volume\") pod \"a7f5ec78-d523-4257-9971-7fc641045a06\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") "
Jul 27 23:48:36.286744 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286733    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdzks\" (UniqueName: \"kubernetes.io/projected/a7f5ec78-d523-4257-9971-7fc641045a06-kube-api-access-jdzks\") pod \"a7f5ec78-d523-4257-9971-7fc641045a06\" (UID: \"a7f5ec78-d523-4257-9971-7fc641045a06\") "
Jul 27 23:48:36.286980 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286757    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-openshift-service-ca\") pod \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\" (UID: \"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e\") "
Jul 27 23:48:36.286980 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286780    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1974feb8-6d42-4d91-91e4-d20f118627eb-plugins-config-volume\") pod \"1974feb8-6d42-4d91-91e4-d20f118627eb\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") "
Jul 27 23:48:36.286980 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.286854    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9znf\" (UniqueName: \"kubernetes.io/projected/1974feb8-6d42-4d91-91e4-d20f118627eb-kube-api-access-c9znf\") pod \"1974feb8-6d42-4d91-91e4-d20f118627eb\" (UID: \"1974feb8-6d42-4d91-91e4-d20f118627eb\") "
Jul 27 23:48:36.287129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.287104    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f5ec78-d523-4257-9971-7fc641045a06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7f5ec78-d523-4257-9971-7fc641045a06" (UID: "a7f5ec78-d523-4257-9971-7fc641045a06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:48:36.287179 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.287159    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" (UID: "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:48:36.287233 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.287190    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f5ec78-d523-4257-9971-7fc641045a06-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:36.287233 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.287222    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1974feb8-6d42-4d91-91e4-d20f118627eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1974feb8-6d42-4d91-91e4-d20f118627eb" (UID: "1974feb8-6d42-4d91-91e4-d20f118627eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:48:36.289136 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.289106    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1974feb8-6d42-4d91-91e4-d20f118627eb-kube-api-access-c9znf" (OuterVolumeSpecName: "kube-api-access-c9znf") pod "1974feb8-6d42-4d91-91e4-d20f118627eb" (UID: "1974feb8-6d42-4d91-91e4-d20f118627eb"). InnerVolumeSpecName "kube-api-access-c9znf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:36.289136 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.289128    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f5ec78-d523-4257-9971-7fc641045a06-kube-api-access-jdzks" (OuterVolumeSpecName: "kube-api-access-jdzks") pod "a7f5ec78-d523-4257-9971-7fc641045a06" (UID: "a7f5ec78-d523-4257-9971-7fc641045a06"). InnerVolumeSpecName "kube-api-access-jdzks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:36.289393 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.289375    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" (UID: "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:48:36.289451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.289434    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-kube-api-access-q6z8f" (OuterVolumeSpecName: "kube-api-access-q6z8f") pod "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" (UID: "2c2ef2c6-caa1-468a-bfb3-e6bb4464163e"). InnerVolumeSpecName "kube-api-access-q6z8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:48:36.387804 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.387773    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:36.387804 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.387805    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdzks\" (UniqueName: \"kubernetes.io/projected/a7f5ec78-d523-4257-9971-7fc641045a06-kube-api-access-jdzks\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:36.387990 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.387817    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:36.387990 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.387826    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1974feb8-6d42-4d91-91e4-d20f118627eb-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:36.387990 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.387834    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9znf\" (UniqueName: \"kubernetes.io/projected/1974feb8-6d42-4d91-91e4-d20f118627eb-kube-api-access-c9znf\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:36.387990 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:36.387842    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6z8f\" (UniqueName: \"kubernetes.io/projected/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e-kube-api-access-q6z8f\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:48:37.115751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.115663    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"
Jul 27 23:48:37.115751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.115689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2" event={"ID":"1974feb8-6d42-4d91-91e4-d20f118627eb","Type":"ContainerDied","Data":"6e90045b01d282826eb5a3903e5b743caae724d35e5e47b093df29f6d2d247b9"}
Jul 27 23:48:37.116232 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.115753    2570 scope.go:117] "RemoveContainer" containerID="ad642e363d125208a3458e156206f95f974ed757de207a90e282842cb346cf35"
Jul 27 23:48:37.117225 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.117199    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph" event={"ID":"a7f5ec78-d523-4257-9971-7fc641045a06","Type":"ContainerDied","Data":"45cbdd005ea9e7977d499b778808b248bc5cff0402a03ac4a055bdda601536bf"}
Jul 27 23:48:37.117340 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.117202    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"
Jul 27 23:48:37.118779 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.118741    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb" event={"ID":"2c2ef2c6-caa1-468a-bfb3-e6bb4464163e","Type":"ContainerDied","Data":"f0d5ff21d47f4206792811b6ed963e013548360848185e58e5e74c6c72fc273e"}
Jul 27 23:48:37.118779 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.118757    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"
Jul 27 23:48:37.124685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.124663    2570 scope.go:117] "RemoveContainer" containerID="b0f7a0c258e309f25aa5479be3788e8b4a2a58790d30e98bccd3bfd25f67dd99"
Jul 27 23:48:37.134390 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.134372    2570 scope.go:117] "RemoveContainer" containerID="0229262636f50abff4ee4fb986474ddf720ba547242badc83e58b698eed694ed"
Jul 27 23:48:37.142883 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.142862    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"]
Jul 27 23:48:37.148547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.148524    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7355578b-77b97df85b-97lt2"]
Jul 27 23:48:37.158846 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.158823    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"]
Jul 27 23:48:37.162131 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.162095    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7355578b-85689f45d4-tbbph"]
Jul 27 23:48:37.173415 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.173389    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"]
Jul 27 23:48:37.177523 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.177500    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7355578b-5bd8c56568-97fpb"]
Jul 27 23:48:37.522942 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.522903    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1974feb8-6d42-4d91-91e4-d20f118627eb" path="/var/lib/kubelet/pods/1974feb8-6d42-4d91-91e4-d20f118627eb/volumes"
Jul 27 23:48:37.523386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.523349    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" path="/var/lib/kubelet/pods/2c2ef2c6-caa1-468a-bfb3-e6bb4464163e/volumes"
Jul 27 23:48:37.523746 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:37.523729    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f5ec78-d523-4257-9971-7fc641045a06" path="/var/lib/kubelet/pods/a7f5ec78-d523-4257-9971-7fc641045a06/volumes"
Jul 27 23:48:40.542187 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:40.542152    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"]
Jul 27 23:48:40.545046 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:40.545016    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7355578b-wfcqm"]
Jul 27 23:48:41.521424 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:41.521395    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55249e52-3c15-4570-9583-c0574cde6ff2" path="/var/lib/kubelet/pods/55249e52-3c15-4570-9583-c0574cde6ff2/volumes"
Jul 27 23:48:42.397233 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397189    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"]
Jul 27 23:48:42.397627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397613    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1974feb8-6d42-4d91-91e4-d20f118627eb" containerName="payload-pre-processing"
Jul 27 23:48:42.397627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397627    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1974feb8-6d42-4d91-91e4-d20f118627eb" containerName="payload-pre-processing"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397638    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397644    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1392df87-900c-4572-ac88-07cfcf017e0f" containerName="istio-proxy"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397654    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f5ec78-d523-4257-9971-7fc641045a06" containerName="payload-processing"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397660    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f5ec78-d523-4257-9971-7fc641045a06" containerName="payload-processing"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397669    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" containerName="maas-api"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397674    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" containerName="maas-api"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397679    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55249e52-3c15-4570-9583-c0574cde6ff2" containerName="revoke-keys"
Jul 27 23:48:42.397703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397684    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="55249e52-3c15-4570-9583-c0574cde6ff2" containerName="revoke-keys"
Jul 27 23:48:42.397936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397744    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1974feb8-6d42-4d91-91e4-d20f118627eb" containerName="payload-pre-processing"
Jul 27 23:48:42.397936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397753    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2ef2c6-caa1-468a-bfb3-e6bb4464163e" containerName="maas-api"
Jul 27 23:48:42.397936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397762    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f5ec78-d523-4257-9971-7fc641045a06" containerName="payload-processing"
Jul 27 23:48:42.397936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.397768    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="55249e52-3c15-4570-9583-c0574cde6ff2" containerName="revoke-keys"
Jul 27 23:48:42.402276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.402254    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.404835 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.404820    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-21351db5-openshift-default-dockercfg-52wwd\""
Jul 27 23:48:42.411368 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.411325    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"]
Jul 27 23:48:42.548810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.548767    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-token\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.548810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.548811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e6eb19-707b-44dd-ac60-88eceabc1daf-istiod-ca-cert\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.548851    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-certs\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.548883    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-envoy\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.548946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-credential-socket\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.549009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-data\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.549038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-socket\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549236 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.549081    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-podinfo\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.549236 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.549097    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx9x\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-kube-api-access-4vx9x\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650173 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-token\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650173 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650123    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e6eb19-707b-44dd-ac60-88eceabc1daf-istiod-ca-cert\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650173 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650158    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-certs\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650192    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-envoy\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650231    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-credential-socket\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650263    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-data\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-socket\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.650957 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650935    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-podinfo\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.651028 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650946    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-data\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.651028 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.650979    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx9x\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-kube-api-access-4vx9x\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.651142 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.651031    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-certs\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.651332 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.651299    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-socket\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.651541 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.651508    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e6eb19-707b-44dd-ac60-88eceabc1daf-istiod-ca-cert\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.652010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.651993    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-credential-socket\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.654238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.654216    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-envoy\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.656545 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.654641    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-podinfo\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.659791 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.659135    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-token\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.660501 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.660481    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx9x\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-kube-api-access-4vx9x\") pod \"e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") " pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.714064 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.714022    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:42.852291 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.852261    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"]
Jul 27 23:48:42.854221 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:42.854198    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e6eb19_707b_44dd_ac60_88eceabc1daf.slice/crio-b7242c9a11046723662caa4216188bdb266b8c977cd8e07f3fd90b911a96e4f6 WatchSource:0}: Error finding container b7242c9a11046723662caa4216188bdb266b8c977cd8e07f3fd90b911a96e4f6: Status 404 returned error can't find the container with id b7242c9a11046723662caa4216188bdb266b8c977cd8e07f3fd90b911a96e4f6
Jul 27 23:48:42.857502 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.857464    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:48:42.857583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.857533    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:48:42.857583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:42.857563    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:48:43.152405 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:43.152368    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" event={"ID":"57e6eb19-707b-44dd-ac60-88eceabc1daf","Type":"ContainerStarted","Data":"1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"}
Jul 27 23:48:43.152611 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:43.152432    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" event={"ID":"57e6eb19-707b-44dd-ac60-88eceabc1daf","Type":"ContainerStarted","Data":"b7242c9a11046723662caa4216188bdb266b8c977cd8e07f3fd90b911a96e4f6"}
Jul 27 23:48:43.173181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:43.173075    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podStartSLOduration=1.173060347 podStartE2EDuration="1.173060347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:43.172085554 +0000 UTC m=+1948.123428957" watchObservedRunningTime="2026-07-27 23:48:43.173060347 +0000 UTC m=+1948.124403734"
Jul 27 23:48:43.714439 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:43.714402    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:43.720176 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:43.720149    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:44.157164 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.157138    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:44.158300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.158284    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:48:44.567468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.567382    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"]
Jul 27 23:48:44.571169 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.571151    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.574006 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.573985    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-21351db5-serving-cert\""
Jul 27 23:48:44.583727 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.583703    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"]
Jul 27 23:48:44.670496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.670458    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc48c02-7414-4e51-b11d-4d02bb662b25-openshift-service-ca\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.670496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.670500    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.670764 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.670522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjdk\" (UniqueName: \"kubernetes.io/projected/8bc48c02-7414-4e51-b11d-4d02bb662b25-kube-api-access-9kjdk\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.771109 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.771071    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc48c02-7414-4e51-b11d-4d02bb662b25-openshift-service-ca\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.771617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.771121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.771617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.771147    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjdk\" (UniqueName: \"kubernetes.io/projected/8bc48c02-7414-4e51-b11d-4d02bb662b25-kube-api-access-9kjdk\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.771617 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:48:44.771294    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-serving-cert: secret "maas-api-e2e-derive-21351db5-serving-cert" not found
Jul 27 23:48:44.771617 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:48:44.771387    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls podName:8bc48c02-7414-4e51-b11d-4d02bb662b25 nodeName:}" failed. No retries permitted until 2026-07-27 23:48:45.271345829 +0000 UTC m=+1950.222689215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls") pod "maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" (UID: "8bc48c02-7414-4e51-b11d-4d02bb662b25") : secret "maas-api-e2e-derive-21351db5-serving-cert" not found
Jul 27 23:48:44.771821 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.771804    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc48c02-7414-4e51-b11d-4d02bb662b25-openshift-service-ca\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:44.781037 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:44.781019    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjdk\" (UniqueName: \"kubernetes.io/projected/8bc48c02-7414-4e51-b11d-4d02bb662b25-kube-api-access-9kjdk\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:45.275093 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.275053    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:45.277674 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.277652    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls\") pod \"maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:45.374303 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.374268    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"]
Jul 27 23:48:45.377797 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.377780    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.380691 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.380662    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-21351db5\""
Jul 27 23:48:45.382347 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.382312    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"]
Jul 27 23:48:45.385982 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.385961    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.388070 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.388047    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"]
Jul 27 23:48:45.388818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.388800    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-21351db5-dockercfg-r7pj9\""
Jul 27 23:48:45.393751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.393728    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"]
Jul 27 23:48:45.477793 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.477503    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.477984 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.477848    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rbr\" (UniqueName: \"kubernetes.io/projected/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-kube-api-access-x2rbr\") pod \"payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.478055 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.477985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-plugins-config-volume\") pod \"payload-processing-e2e-derive-21351db5-675bd6f78-8nl76\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") " pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.478055 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.478016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwlgj\" (UniqueName: \"kubernetes.io/projected/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-kube-api-access-qwlgj\") pod \"payload-processing-e2e-derive-21351db5-675bd6f78-8nl76\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") " pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.483068 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.483041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:45.579173 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.579138    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.579322 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.579192    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rbr\" (UniqueName: \"kubernetes.io/projected/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-kube-api-access-x2rbr\") pod \"payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.579322 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.579289    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-plugins-config-volume\") pod \"payload-processing-e2e-derive-21351db5-675bd6f78-8nl76\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") " pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.579322 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.579318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwlgj\" (UniqueName: \"kubernetes.io/projected/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-kube-api-access-qwlgj\") pod \"payload-processing-e2e-derive-21351db5-675bd6f78-8nl76\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") " pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.580054 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.580033    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.580311 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.580270    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-plugins-config-volume\") pod \"payload-processing-e2e-derive-21351db5-675bd6f78-8nl76\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") " pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.588298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.588248    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwlgj\" (UniqueName: \"kubernetes.io/projected/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-kube-api-access-qwlgj\") pod \"payload-processing-e2e-derive-21351db5-675bd6f78-8nl76\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") " pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.588430 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.588408    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rbr\" (UniqueName: \"kubernetes.io/projected/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-kube-api-access-x2rbr\") pod \"payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.627225 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.627203    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"]
Jul 27 23:48:45.629328 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:45.629302    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc48c02_7414_4e51_b11d_4d02bb662b25.slice/crio-3386a502fd7a4f132eaea33e368875b2513357a801c16cbfbd8600ad3d575004 WatchSource:0}: Error finding container 3386a502fd7a4f132eaea33e368875b2513357a801c16cbfbd8600ad3d575004: Status 404 returned error can't find the container with id 3386a502fd7a4f132eaea33e368875b2513357a801c16cbfbd8600ad3d575004
Jul 27 23:48:45.690538 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.690503    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:45.700894 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.700866    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:45.840638 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:45.840605    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdbcfe7_4a54_4f72_8ef3_d6e25a3c3e27.slice/crio-21d05b882d21489157bf13021875d489688a47172d802a89fdaf6658846dc182 WatchSource:0}: Error finding container 21d05b882d21489157bf13021875d489688a47172d802a89fdaf6658846dc182: Status 404 returned error can't find the container with id 21d05b882d21489157bf13021875d489688a47172d802a89fdaf6658846dc182
Jul 27 23:48:45.840963 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.840882    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"]
Jul 27 23:48:45.864918 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:45.864836    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"]
Jul 27 23:48:45.867205 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:48:45.867168    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3bea09_b68c_4f2e_a05a_b1869b8336d4.slice/crio-e3d7b94379b7dca17def82bac9496aaf9443bf252dc08e23bc67c4d2f8c45de3 WatchSource:0}: Error finding container e3d7b94379b7dca17def82bac9496aaf9443bf252dc08e23bc67c4d2f8c45de3: Status 404 returned error can't find the container with id e3d7b94379b7dca17def82bac9496aaf9443bf252dc08e23bc67c4d2f8c45de3
Jul 27 23:48:46.170742 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.170697    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76" event={"ID":"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27","Type":"ContainerStarted","Data":"83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617"}
Jul 27 23:48:46.170909 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.170810    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76" event={"ID":"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27","Type":"ContainerStarted","Data":"21d05b882d21489157bf13021875d489688a47172d802a89fdaf6658846dc182"}
Jul 27 23:48:46.171007 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.170980    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:46.172778 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.172747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss" event={"ID":"3a3bea09-b68c-4f2e-a05a-b1869b8336d4","Type":"ContainerStarted","Data":"a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a"}
Jul 27 23:48:46.172890 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.172793    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss" event={"ID":"3a3bea09-b68c-4f2e-a05a-b1869b8336d4","Type":"ContainerStarted","Data":"e3d7b94379b7dca17def82bac9496aaf9443bf252dc08e23bc67c4d2f8c45de3"}
Jul 27 23:48:46.172890 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.172836    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:46.174478 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.174455    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" event={"ID":"8bc48c02-7414-4e51-b11d-4d02bb662b25","Type":"ContainerStarted","Data":"b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b"}
Jul 27 23:48:46.174579 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.174507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" event={"ID":"8bc48c02-7414-4e51-b11d-4d02bb662b25","Type":"ContainerStarted","Data":"3386a502fd7a4f132eaea33e368875b2513357a801c16cbfbd8600ad3d575004"}
Jul 27 23:48:46.174579 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.174543    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:46.186679 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.186636    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76" podStartSLOduration=1.186621156 podStartE2EDuration="1.186621156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:46.18398273 +0000 UTC m=+1951.135326144" watchObservedRunningTime="2026-07-27 23:48:46.186621156 +0000 UTC m=+1951.137964547"
Jul 27 23:48:46.200538 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.200493    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" podStartSLOduration=1.739063896 podStartE2EDuration="2.200474529s" totalImagesPullingTime="461.410633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:46.198405128 +0000 UTC m=+1951.149748541" watchObservedRunningTime="2026-07-27 23:48:46.200474529 +0000 UTC m=+1951.151817916"
Jul 27 23:48:46.214845 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:46.214803    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss" podStartSLOduration=1.2147896 podStartE2EDuration="1.2147896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:48:46.213384328 +0000 UTC m=+1951.164727734" watchObservedRunningTime="2026-07-27 23:48:46.2147896 +0000 UTC m=+1951.166132986"
Jul 27 23:48:52.185200 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:52.185169    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:48:57.182296 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:57.182256    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:48:57.182843 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:57.182624    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:48:59.158849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.158812    2570 patch_prober.go:28] interesting pod/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 23:48:59.159280 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.158884    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 23:48:59.245425 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.245392    2570 generic.go:356] "Generic (PLEG): container finished" podID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerID="1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5" exitCode=0
Jul 27 23:48:59.245607 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.245470    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" event={"ID":"57e6eb19-707b-44dd-ac60-88eceabc1daf","Type":"ContainerDied","Data":"1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"}
Jul 27 23:48:59.245985 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.245968    2570 scope.go:117] "RemoveContainer" containerID="1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"
Jul 27 23:48:59.247033 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.247006    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:48:59.247098 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.247070    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:48:59.247140 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:48:59.247112    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:49:00.253932 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:00.252940    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" event={"ID":"57e6eb19-707b-44dd-ac60-88eceabc1daf","Type":"ContainerStarted","Data":"f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9"}
Jul 27 23:49:00.714968 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:00.714923    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:49:00.715242 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:00.715219    2570 patch_prober.go:28] interesting pod/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 23:49:00.715291 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:00.715270    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 23:49:01.715203 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:01.715164    2570 patch_prober.go:28] interesting pod/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 23:49:01.715649 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:01.715222    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 23:49:02.714663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:02.714622    2570 patch_prober.go:28] interesting pod/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 23:49:02.714871 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:02.714685    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 23:49:03.715089 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:03.715052    2570 patch_prober.go:28] interesting pod/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 23:49:03.715487 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:03.715123    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 23:49:04.718254 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:04.718225    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:49:04.718708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:04.718593    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:49:04.719201 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:04.719187    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:49:19.706200 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:19.706166    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"]
Jul 27 23:49:19.707033 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:19.706993    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" podUID="8bc48c02-7414-4e51-b11d-4d02bb662b25" containerName="maas-api" containerID="cri-o://b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b" gracePeriod=30
Jul 27 23:49:19.754290 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:19.754252    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"]
Jul 27 23:49:19.754560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:19.754531    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76" podUID="8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" containerName="payload-processing" containerID="cri-o://83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617" gracePeriod=30
Jul 27 23:49:19.763953 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:19.763925    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"]
Jul 27 23:49:19.764194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:19.764163    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss" podUID="3a3bea09-b68c-4f2e-a05a-b1869b8336d4" containerName="payload-pre-processing" containerID="cri-o://a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a" gracePeriod=30
Jul 27 23:49:20.018259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.018203    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:49:20.065374 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.065330    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:49:20.068600 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.068581    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:49:20.214208 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214173    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kjdk\" (UniqueName: \"kubernetes.io/projected/8bc48c02-7414-4e51-b11d-4d02bb662b25-kube-api-access-9kjdk\") pod \"8bc48c02-7414-4e51-b11d-4d02bb662b25\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") "
Jul 27 23:49:20.214208 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214215    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2rbr\" (UniqueName: \"kubernetes.io/projected/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-kube-api-access-x2rbr\") pod \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") "
Jul 27 23:49:20.214491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214237    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc48c02-7414-4e51-b11d-4d02bb662b25-openshift-service-ca\") pod \"8bc48c02-7414-4e51-b11d-4d02bb662b25\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") "
Jul 27 23:49:20.214491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214275    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls\") pod \"8bc48c02-7414-4e51-b11d-4d02bb662b25\" (UID: \"8bc48c02-7414-4e51-b11d-4d02bb662b25\") "
Jul 27 23:49:20.214491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214302    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-plugins-config-volume\") pod \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\" (UID: \"3a3bea09-b68c-4f2e-a05a-b1869b8336d4\") "
Jul 27 23:49:20.214491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214318    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-plugins-config-volume\") pod \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") "
Jul 27 23:49:20.214491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214335    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwlgj\" (UniqueName: \"kubernetes.io/projected/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-kube-api-access-qwlgj\") pod \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\" (UID: \"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27\") "
Jul 27 23:49:20.214829 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214787    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a3bea09-b68c-4f2e-a05a-b1869b8336d4" (UID: "3a3bea09-b68c-4f2e-a05a-b1869b8336d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:49:20.214890 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214860    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc48c02-7414-4e51-b11d-4d02bb662b25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bc48c02-7414-4e51-b11d-4d02bb662b25" (UID: "8bc48c02-7414-4e51-b11d-4d02bb662b25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:49:20.214954 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.214910    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" (UID: "8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:49:20.216606 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.216583    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bc48c02-7414-4e51-b11d-4d02bb662b25" (UID: "8bc48c02-7414-4e51-b11d-4d02bb662b25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:49:20.216787 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.216763    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-kube-api-access-qwlgj" (OuterVolumeSpecName: "kube-api-access-qwlgj") pod "8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" (UID: "8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27"). InnerVolumeSpecName "kube-api-access-qwlgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:49:20.217041 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.217025    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc48c02-7414-4e51-b11d-4d02bb662b25-kube-api-access-9kjdk" (OuterVolumeSpecName: "kube-api-access-9kjdk") pod "8bc48c02-7414-4e51-b11d-4d02bb662b25" (UID: "8bc48c02-7414-4e51-b11d-4d02bb662b25"). InnerVolumeSpecName "kube-api-access-9kjdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:49:20.217084 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.217062    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-kube-api-access-x2rbr" (OuterVolumeSpecName: "kube-api-access-x2rbr") pod "3a3bea09-b68c-4f2e-a05a-b1869b8336d4" (UID: "3a3bea09-b68c-4f2e-a05a-b1869b8336d4"). InnerVolumeSpecName "kube-api-access-x2rbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315724    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjdk\" (UniqueName: \"kubernetes.io/projected/8bc48c02-7414-4e51-b11d-4d02bb662b25-kube-api-access-9kjdk\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315753    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rbr\" (UniqueName: \"kubernetes.io/projected/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-kube-api-access-x2rbr\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315763    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc48c02-7414-4e51-b11d-4d02bb662b25-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315773    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc48c02-7414-4e51-b11d-4d02bb662b25-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315783    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3bea09-b68c-4f2e-a05a-b1869b8336d4-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315791    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.315810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.315799    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwlgj\" (UniqueName: \"kubernetes.io/projected/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27-kube-api-access-qwlgj\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:20.365920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.365882    2570 generic.go:356] "Generic (PLEG): container finished" podID="8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" containerID="83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617" exitCode=0
Jul 27 23:49:20.366065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.365931    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"
Jul 27 23:49:20.366065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.365959    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76" event={"ID":"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27","Type":"ContainerDied","Data":"83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617"}
Jul 27 23:49:20.366065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.366013    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76" event={"ID":"8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27","Type":"ContainerDied","Data":"21d05b882d21489157bf13021875d489688a47172d802a89fdaf6658846dc182"}
Jul 27 23:49:20.366065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.366035    2570 scope.go:117] "RemoveContainer" containerID="83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617"
Jul 27 23:49:20.367314 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.367294    2570 generic.go:356] "Generic (PLEG): container finished" podID="3a3bea09-b68c-4f2e-a05a-b1869b8336d4" containerID="a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a" exitCode=0
Jul 27 23:49:20.367450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.367332    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"
Jul 27 23:49:20.367450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.367383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss" event={"ID":"3a3bea09-b68c-4f2e-a05a-b1869b8336d4","Type":"ContainerDied","Data":"a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a"}
Jul 27 23:49:20.367450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.367425    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss" event={"ID":"3a3bea09-b68c-4f2e-a05a-b1869b8336d4","Type":"ContainerDied","Data":"e3d7b94379b7dca17def82bac9496aaf9443bf252dc08e23bc67c4d2f8c45de3"}
Jul 27 23:49:20.369029 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.369003    2570 generic.go:356] "Generic (PLEG): container finished" podID="8bc48c02-7414-4e51-b11d-4d02bb662b25" containerID="b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b" exitCode=0
Jul 27 23:49:20.369165 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.369047    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" event={"ID":"8bc48c02-7414-4e51-b11d-4d02bb662b25","Type":"ContainerDied","Data":"b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b"}
Jul 27 23:49:20.369165 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.369055    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"
Jul 27 23:49:20.369165 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.369086    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx" event={"ID":"8bc48c02-7414-4e51-b11d-4d02bb662b25","Type":"ContainerDied","Data":"3386a502fd7a4f132eaea33e368875b2513357a801c16cbfbd8600ad3d575004"}
Jul 27 23:49:20.376139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.376119    2570 scope.go:117] "RemoveContainer" containerID="83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617"
Jul 27 23:49:20.376541 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:49:20.376514    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617\": container with ID starting with 83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617 not found: ID does not exist" containerID="83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617"
Jul 27 23:49:20.376652 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.376553    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617"} err="failed to get container status \"83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617\": rpc error: code = NotFound desc = could not find container \"83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617\": container with ID starting with 83749e2516e61700b64985280eba5f09cf34b140b6374d6e2a57b58c8af8c617 not found: ID does not exist"
Jul 27 23:49:20.376652 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.376578    2570 scope.go:117] "RemoveContainer" containerID="a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a"
Jul 27 23:49:20.385982 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.385961    2570 scope.go:117] "RemoveContainer" containerID="a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a"
Jul 27 23:49:20.386257 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:49:20.386236    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a\": container with ID starting with a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a not found: ID does not exist" containerID="a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a"
Jul 27 23:49:20.386308 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.386269    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a"} err="failed to get container status \"a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a\": rpc error: code = NotFound desc = could not find container \"a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a\": container with ID starting with a500e3dd8955dc5563c316fa9e07e89d781e74c0f2a8e0bb19f4f3bb19a0b37a not found: ID does not exist"
Jul 27 23:49:20.386308 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.386286    2570 scope.go:117] "RemoveContainer" containerID="b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b"
Jul 27 23:49:20.393228 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.393203    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"]
Jul 27 23:49:20.394400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.394348    2570 scope.go:117] "RemoveContainer" containerID="b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b"
Jul 27 23:49:20.394846 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:49:20.394818    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b\": container with ID starting with b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b not found: ID does not exist" containerID="b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b"
Jul 27 23:49:20.394945 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.394858    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b"} err="failed to get container status \"b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b\": rpc error: code = NotFound desc = could not find container \"b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b\": container with ID starting with b71d5928e6ab38889a72b00b3dc942448d645165f3266909aa35a7402e97427b not found: ID does not exist"
Jul 27 23:49:20.396973 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.396947    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-21351db5-7d5ff587b8-wjrss"]
Jul 27 23:49:20.408139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.408097    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"]
Jul 27 23:49:20.410502 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.410478    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-21351db5-675bd6f78-8nl76"]
Jul 27 23:49:20.421223 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.421189    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"]
Jul 27 23:49:20.425377 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:20.425332    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-21351db5-77ccc9964b-zwjpx"]
Jul 27 23:49:21.523169 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:21.523132    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3bea09-b68c-4f2e-a05a-b1869b8336d4" path="/var/lib/kubelet/pods/3a3bea09-b68c-4f2e-a05a-b1869b8336d4/volumes"
Jul 27 23:49:21.523605 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:21.523542    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bc48c02-7414-4e51-b11d-4d02bb662b25" path="/var/lib/kubelet/pods/8bc48c02-7414-4e51-b11d-4d02bb662b25/volumes"
Jul 27 23:49:21.523851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:21.523838    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" path="/var/lib/kubelet/pods/8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27/volumes"
Jul 27 23:49:24.547558 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:24.547526    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"]
Jul 27 23:49:24.547987 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:24.547761    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy" containerID="cri-o://f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9" gracePeriod=30
Jul 27 23:49:29.799562 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.799539    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:49:29.903850 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.903821    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-data\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.903864    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-envoy\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.903918    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-token\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.903934    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-certs\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.903954    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-socket\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904010 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.903998    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vx9x\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-kube-api-access-4vx9x\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904057    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-podinfo\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-credential-socket\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904103    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e6eb19-707b-44dd-ac60-88eceabc1daf-istiod-ca-cert\") pod \"57e6eb19-707b-44dd-ac60-88eceabc1daf\" (UID: \"57e6eb19-707b-44dd-ac60-88eceabc1daf\") "
Jul 27 23:49:29.904292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904239    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:49:29.904292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904252    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:49:29.904587 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904399    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:29.904587 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904456    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:29.904587 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904489    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:49:29.904911 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.904591    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e6eb19-707b-44dd-ac60-88eceabc1daf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:49:29.905288 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.905259    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-data" (OuterVolumeSpecName: "istio-data") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:49:29.906386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.906326    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:49:29.906386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.906340    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:49:29.906537 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.906405    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-kube-api-access-4vx9x" (OuterVolumeSpecName: "kube-api-access-4vx9x") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "kube-api-access-4vx9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:49:29.906537 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:29.906405    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-token" (OuterVolumeSpecName: "istio-token") pod "57e6eb19-707b-44dd-ac60-88eceabc1daf" (UID: "57e6eb19-707b-44dd-ac60-88eceabc1daf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:49:30.005238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005187    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.005238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005243    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.005496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005269    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e6eb19-707b-44dd-ac60-88eceabc1daf-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.005496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005279    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.005496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005286    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.005496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005294    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.005496 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.005302    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx9x\" (UniqueName: \"kubernetes.io/projected/57e6eb19-707b-44dd-ac60-88eceabc1daf-kube-api-access-4vx9x\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:49:30.429587 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.429557    2570 generic.go:356] "Generic (PLEG): container finished" podID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerID="f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9" exitCode=0
Jul 27 23:49:30.429796 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.429607    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"
Jul 27 23:49:30.429796 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.429655    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" event={"ID":"57e6eb19-707b-44dd-ac60-88eceabc1daf","Type":"ContainerDied","Data":"f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9"}
Jul 27 23:49:30.429796 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.429723    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd" event={"ID":"57e6eb19-707b-44dd-ac60-88eceabc1daf","Type":"ContainerDied","Data":"b7242c9a11046723662caa4216188bdb266b8c977cd8e07f3fd90b911a96e4f6"}
Jul 27 23:49:30.429796 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.429756    2570 scope.go:117] "RemoveContainer" containerID="f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9"
Jul 27 23:49:30.439305 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.439289    2570 scope.go:117] "RemoveContainer" containerID="1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"
Jul 27 23:49:30.446916 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.446900    2570 scope.go:117] "RemoveContainer" containerID="f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9"
Jul 27 23:49:30.447185 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:49:30.447166    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9\": container with ID starting with f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9 not found: ID does not exist" containerID="f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9"
Jul 27 23:49:30.447231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.447195    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9"} err="failed to get container status \"f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9\": rpc error: code = NotFound desc = could not find container \"f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9\": container with ID starting with f19eff1b4d6b55a398ee0fc64f6c759870bfb2249b60261ab2ee4e6bbd8df1d9 not found: ID does not exist"
Jul 27 23:49:30.447231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.447213    2570 scope.go:117] "RemoveContainer" containerID="1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"
Jul 27 23:49:30.447481 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:49:30.447464    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5\": container with ID starting with 1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5 not found: ID does not exist" containerID="1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"
Jul 27 23:49:30.447539 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.447486    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5"} err="failed to get container status \"1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5\": rpc error: code = NotFound desc = could not find container \"1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5\": container with ID starting with 1cd99642ab56fab3481096085080a465490141259919337982c18c8f32a7b3d5 not found: ID does not exist"
Jul 27 23:49:30.452336 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.452316    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"]
Jul 27 23:49:30.453929 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:30.453910    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-21351db5-openshift-default-9c879cbdd-vsqdd"]
Jul 27 23:49:31.521525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:49:31.521495    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" path="/var/lib/kubelet/pods/57e6eb19-707b-44dd-ac60-88eceabc1daf/volumes"
Jul 27 23:51:10.822148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822115    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"]
Jul 27 23:51:10.822751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822658    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3bea09-b68c-4f2e-a05a-b1869b8336d4" containerName="payload-pre-processing"
Jul 27 23:51:10.822751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822677    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3bea09-b68c-4f2e-a05a-b1869b8336d4" containerName="payload-pre-processing"
Jul 27 23:51:10.822751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822709    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy"
Jul 27 23:51:10.822751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822718    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy"
Jul 27 23:51:10.822751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822738    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" containerName="payload-processing"
Jul 27 23:51:10.822751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822746    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" containerName="payload-processing"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822771    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bc48c02-7414-4e51-b11d-4d02bb662b25" containerName="maas-api"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822779    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bc48c02-7414-4e51-b11d-4d02bb662b25" containerName="maas-api"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822863    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdbcfe7-4a54-4f72-8ef3-d6e25a3c3e27" containerName="payload-processing"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822875    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822884    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bc48c02-7414-4e51-b11d-4d02bb662b25" containerName="maas-api"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822893    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3bea09-b68c-4f2e-a05a-b1869b8336d4" containerName="payload-pre-processing"
Jul 27 23:51:10.823078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.822904    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy"
Jul 27 23:51:10.825447 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.825425    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.828175 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.828154    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0050c7fc-openshift-default-dockercfg-5k2c4\""
Jul 27 23:51:10.849282 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.849254    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"]
Jul 27 23:51:10.897800 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.897766    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-data\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.897966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.897829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-socket\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.897966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.897854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69aec922-4d70-4c61-851a-f7d213f077e0-istiod-ca-cert\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.897966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.897882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-certs\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.898075 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.897967    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9n9k\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-kube-api-access-h9n9k\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.898075 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.898009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-envoy\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.898075 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.898036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69aec922-4d70-4c61-851a-f7d213f077e0-istio-podinfo\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.898075 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.898065    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-credential-socket\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.898199 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.898080    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-istio-token\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998784 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998746    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-data\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998804    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-socket\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69aec922-4d70-4c61-851a-f7d213f077e0-istiod-ca-cert\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-certs\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9n9k\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-kube-api-access-h9n9k\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998909    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-envoy\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998927    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69aec922-4d70-4c61-851a-f7d213f077e0-istio-podinfo\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.998978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998967    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-credential-socket\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.999343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.998988    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-istio-token\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.999343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.999288    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-data\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.999343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.999322    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-socket\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.999539 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.999518    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-credential-socket\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.999627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.999610    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-certs\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:10.999686 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:10.999659    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69aec922-4d70-4c61-851a-f7d213f077e0-istiod-ca-cert\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:11.001525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.001505    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-envoy\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:11.001637 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.001625    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69aec922-4d70-4c61-851a-f7d213f077e0-istio-podinfo\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:11.007018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.006987    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-istio-token\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:11.007137 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.007036    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9n9k\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-kube-api-access-h9n9k\") pod \"e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") " pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:11.136043 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.136014    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:11.274455 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.274429    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"]
Jul 27 23:51:11.275809 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:51:11.275774    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69aec922_4d70_4c61_851a_f7d213f077e0.slice/crio-de734943910951b5eedac4a9bfaeb06e58934044827c1d2d19b80a9269861d33 WatchSource:0}: Error finding container de734943910951b5eedac4a9bfaeb06e58934044827c1d2d19b80a9269861d33: Status 404 returned error can't find the container with id de734943910951b5eedac4a9bfaeb06e58934044827c1d2d19b80a9269861d33
Jul 27 23:51:11.278094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.278056    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:51:11.278175 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.278147    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:51:11.278217 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.278188    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:51:11.928074 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.928043    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" event={"ID":"69aec922-4d70-4c61-851a-f7d213f077e0","Type":"ContainerStarted","Data":"fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"}
Jul 27 23:51:11.928593 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.928125    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" event={"ID":"69aec922-4d70-4c61-851a-f7d213f077e0","Type":"ContainerStarted","Data":"de734943910951b5eedac4a9bfaeb06e58934044827c1d2d19b80a9269861d33"}
Jul 27 23:51:11.950203 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:11.950147    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podStartSLOduration=1.950131807 podStartE2EDuration="1.950131807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:51:11.948262857 +0000 UTC m=+2096.899606302" watchObservedRunningTime="2026-07-27 23:51:11.950131807 +0000 UTC m=+2096.901475263"
Jul 27 23:51:12.136427 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:12.136387    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:12.141565 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:12.141539    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:12.932426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:12.932391    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:12.933316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:12.933298    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:13.927740 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:13.927705    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"]
Jul 27 23:51:13.928159 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:13.928144    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy"
Jul 27 23:51:13.928215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:13.928162    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e6eb19-707b-44dd-ac60-88eceabc1daf" containerName="istio-proxy"
Jul 27 23:51:13.931383 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:13.931346    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:13.933866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:13.933836    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0050c7fc-serving-cert\""
Jul 27 23:51:13.941580 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:13.941549    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"]
Jul 27 23:51:14.028876 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.028842    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4k8\" (UniqueName: \"kubernetes.io/projected/554e2752-4244-4f10-9c3b-21885cabba78-kube-api-access-fl4k8\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.029066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.029013    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.029250 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.029227    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554e2752-4244-4f10-9c3b-21885cabba78-openshift-service-ca\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.130231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.130195    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4k8\" (UniqueName: \"kubernetes.io/projected/554e2752-4244-4f10-9c3b-21885cabba78-kube-api-access-fl4k8\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.130231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.130233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.130520 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.130273    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554e2752-4244-4f10-9c3b-21885cabba78-openshift-service-ca\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.130520 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:51:14.130384    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-serving-cert: secret "maas-api-e2e-mt-0050c7fc-serving-cert" not found
Jul 27 23:51:14.130520 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:51:14.130458    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls podName:554e2752-4244-4f10-9c3b-21885cabba78 nodeName:}" failed. No retries permitted until 2026-07-27 23:51:14.630431976 +0000 UTC m=+2099.581775340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls") pod "maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" (UID: "554e2752-4244-4f10-9c3b-21885cabba78") : secret "maas-api-e2e-mt-0050c7fc-serving-cert" not found
Jul 27 23:51:14.130934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.130916    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554e2752-4244-4f10-9c3b-21885cabba78-openshift-service-ca\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.146090 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.146057    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4k8\" (UniqueName: \"kubernetes.io/projected/554e2752-4244-4f10-9c3b-21885cabba78-kube-api-access-fl4k8\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.635266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.635224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.638333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.638306    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls\") pod \"maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.727657 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.727593    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"]
Jul 27 23:51:14.731414 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.731391    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.736014 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.735988    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0050c7fc\""
Jul 27 23:51:14.736131 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.736020    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0050c7fc-dockercfg-q9zmd\""
Jul 27 23:51:14.745310 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.745283    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"]
Jul 27 23:51:14.765439 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.765409    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"]
Jul 27 23:51:14.768746 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.768728    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.779176 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.779152    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"]
Jul 27 23:51:14.837391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.837325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-plugins-config-volume\") pod \"payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") " pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.837602 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.837438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrwrl\" (UniqueName: \"kubernetes.io/projected/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-kube-api-access-lrwrl\") pod \"payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") " pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.837602 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.837504    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvmv\" (UniqueName: \"kubernetes.io/projected/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-kube-api-access-pfvmv\") pod \"payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.837602 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.837557    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.845181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.845153    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:14.939118 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.939068    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrwrl\" (UniqueName: \"kubernetes.io/projected/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-kube-api-access-lrwrl\") pod \"payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") " pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.939574 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.939156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvmv\" (UniqueName: \"kubernetes.io/projected/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-kube-api-access-pfvmv\") pod \"payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.939574 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.939211    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.939574 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.939271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-plugins-config-volume\") pod \"payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") " pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.940258 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.940233    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.940425 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.940272    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-plugins-config-volume\") pod \"payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") " pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.948791 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.948673    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvmv\" (UniqueName: \"kubernetes.io/projected/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-kube-api-access-pfvmv\") pod \"payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:14.949095 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.949069    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrwrl\" (UniqueName: \"kubernetes.io/projected/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-kube-api-access-lrwrl\") pod \"payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") " pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:14.987238 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.987208    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"]
Jul 27 23:51:14.988459 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:51:14.988430    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554e2752_4244_4f10_9c3b_21885cabba78.slice/crio-f4c9e65725d73c9890bb0bc7a2a5b2e6f4adbb03570d84afbfda2f65796c622f WatchSource:0}: Error finding container f4c9e65725d73c9890bb0bc7a2a5b2e6f4adbb03570d84afbfda2f65796c622f: Status 404 returned error can't find the container with id f4c9e65725d73c9890bb0bc7a2a5b2e6f4adbb03570d84afbfda2f65796c622f
Jul 27 23:51:14.989913 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:14.989895    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:51:15.042601 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.042571    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:15.081077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.081046    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:15.194855 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.194824    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"]
Jul 27 23:51:15.196956 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:51:15.196920    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ab7f79_bc0d_4598_8a07_3ea5788d57dc.slice/crio-ce25eec35c0d98098b3feab768e63391ddb4f6408ff0ea8a95bdc5dfe8f5a3a8 WatchSource:0}: Error finding container ce25eec35c0d98098b3feab768e63391ddb4f6408ff0ea8a95bdc5dfe8f5a3a8: Status 404 returned error can't find the container with id ce25eec35c0d98098b3feab768e63391ddb4f6408ff0ea8a95bdc5dfe8f5a3a8
Jul 27 23:51:15.238596 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.238545    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"]
Jul 27 23:51:15.240626 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:51:15.240598    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ed64a1_3db5_4dc7_a2b4_6a14f43e2278.slice/crio-3bcc87de946e87cc251924fdc6de6eeab0f8cce4558f6c5cf5337e14b8e1670f WatchSource:0}: Error finding container 3bcc87de946e87cc251924fdc6de6eeab0f8cce4558f6c5cf5337e14b8e1670f: Status 404 returned error can't find the container with id 3bcc87de946e87cc251924fdc6de6eeab0f8cce4558f6c5cf5337e14b8e1670f
Jul 27 23:51:15.546588 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.546559    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:51:15.548819 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.548793    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:51:15.953722 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.953679    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz" event={"ID":"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278","Type":"ContainerStarted","Data":"d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e"}
Jul 27 23:51:15.954251 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.953746    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz" event={"ID":"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278","Type":"ContainerStarted","Data":"3bcc87de946e87cc251924fdc6de6eeab0f8cce4558f6c5cf5337e14b8e1670f"}
Jul 27 23:51:15.954251 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.953859    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:15.955818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.955790    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6" event={"ID":"23ab7f79-bc0d-4598-8a07-3ea5788d57dc","Type":"ContainerStarted","Data":"0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87"}
Jul 27 23:51:15.955961 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.955842    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6" event={"ID":"23ab7f79-bc0d-4598-8a07-3ea5788d57dc","Type":"ContainerStarted","Data":"ce25eec35c0d98098b3feab768e63391ddb4f6408ff0ea8a95bdc5dfe8f5a3a8"}
Jul 27 23:51:15.955961 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.955883    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:15.957866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.957844    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" event={"ID":"554e2752-4244-4f10-9c3b-21885cabba78","Type":"ContainerStarted","Data":"d2ad89cc4ba11ea668da0d6f804334b23e961900f7dcda790112f3601f3027bb"}
Jul 27 23:51:15.957943 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.957886    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" event={"ID":"554e2752-4244-4f10-9c3b-21885cabba78","Type":"ContainerStarted","Data":"f4c9e65725d73c9890bb0bc7a2a5b2e6f4adbb03570d84afbfda2f65796c622f"}
Jul 27 23:51:15.958165 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.958116    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:15.969933 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.969883    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz" podStartSLOduration=1.969865067 podStartE2EDuration="1.969865067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:51:15.968070159 +0000 UTC m=+2100.919413548" watchObservedRunningTime="2026-07-27 23:51:15.969865067 +0000 UTC m=+2100.921208462"
Jul 27 23:51:15.987498 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:15.987441    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" podStartSLOduration=2.575747196 podStartE2EDuration="2.987419262s" totalImagesPullingTime="411.672066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:51:15.984810338 +0000 UTC m=+2100.936153723" watchObservedRunningTime="2026-07-27 23:51:15.987419262 +0000 UTC m=+2100.938762641"
Jul 27 23:51:16.019558 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:16.019503    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6" podStartSLOduration=2.019484301 podStartE2EDuration="2.019484301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:51:16.018097886 +0000 UTC m=+2100.969441291" watchObservedRunningTime="2026-07-27 23:51:16.019484301 +0000 UTC m=+2100.970827688"
Jul 27 23:51:21.977017 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:21.976986    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:26.965085 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:26.965050    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:26.965554 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:26.965241    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:28.013052 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:28.012299    2570 patch_prober.go:28] interesting pod/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:48814->10.133.0.53:15021: read: connection reset by peer" start-of-body=
Jul 27 23:51:28.013052 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:28.012424    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:48814->10.133.0.53:15021: read: connection reset by peer"
Jul 27 23:51:29.033884 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:29.033846    2570 generic.go:356] "Generic (PLEG): container finished" podID="69aec922-4d70-4c61-851a-f7d213f077e0" containerID="fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb" exitCode=0
Jul 27 23:51:29.034314 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:29.033923    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" event={"ID":"69aec922-4d70-4c61-851a-f7d213f077e0","Type":"ContainerDied","Data":"fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"}
Jul 27 23:51:29.034410 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:29.034311    2570 scope.go:117] "RemoveContainer" containerID="fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"
Jul 27 23:51:29.035313 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:29.035283    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:51:29.035464 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:29.035385    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:51:29.035464 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:29.035439    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:51:30.041158 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:30.041119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" event={"ID":"69aec922-4d70-4c61-851a-f7d213f077e0","Type":"ContainerStarted","Data":"8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b"}
Jul 27 23:51:30.137217 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:30.137177    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:30.137481 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:30.137458    2570 patch_prober.go:28] interesting pod/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 27 23:51:30.137544 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:30.137525    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 27 23:51:31.137374 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:31.137319    2570 patch_prober.go:28] interesting pod/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 27 23:51:31.137790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:31.137399    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 27 23:51:32.137189 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:32.137149    2570 patch_prober.go:28] interesting pod/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 27 23:51:32.137402 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:32.137210    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 27 23:51:33.136986 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:33.136947    2570 patch_prober.go:28] interesting pod/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 27 23:51:33.137181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:33.137004    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 27 23:51:34.140081 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:34.140051    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:34.140490 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:34.140392    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:34.141005 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:34.140984    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:35.964047 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:35.964015    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"]
Jul 27 23:51:35.964550 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:35.964275    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" podUID="554e2752-4244-4f10-9c3b-21885cabba78" containerName="maas-api" containerID="cri-o://d2ad89cc4ba11ea668da0d6f804334b23e961900f7dcda790112f3601f3027bb" gracePeriod=30
Jul 27 23:51:36.017431 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.017398    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"]
Jul 27 23:51:36.017705 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.017675    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz" podUID="56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" containerName="payload-processing" containerID="cri-o://d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e" gracePeriod=30
Jul 27 23:51:36.023934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.023904    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"]
Jul 27 23:51:36.024158 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.024134    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6" podUID="23ab7f79-bc0d-4598-8a07-3ea5788d57dc" containerName="payload-pre-processing" containerID="cri-o://0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87" gracePeriod=30
Jul 27 23:51:36.084161 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.084131    2570 generic.go:356] "Generic (PLEG): container finished" podID="554e2752-4244-4f10-9c3b-21885cabba78" containerID="d2ad89cc4ba11ea668da0d6f804334b23e961900f7dcda790112f3601f3027bb" exitCode=0
Jul 27 23:51:36.084333 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.084204    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" event={"ID":"554e2752-4244-4f10-9c3b-21885cabba78","Type":"ContainerDied","Data":"d2ad89cc4ba11ea668da0d6f804334b23e961900f7dcda790112f3601f3027bb"}
Jul 27 23:51:36.285111 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.285084    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:36.327810 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.327786    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:36.331698 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.331677    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:36.343156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.343128    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl4k8\" (UniqueName: \"kubernetes.io/projected/554e2752-4244-4f10-9c3b-21885cabba78-kube-api-access-fl4k8\") pod \"554e2752-4244-4f10-9c3b-21885cabba78\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") "
Jul 27 23:51:36.343300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.343202    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls\") pod \"554e2752-4244-4f10-9c3b-21885cabba78\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") "
Jul 27 23:51:36.343300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.343316    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554e2752-4244-4f10-9c3b-21885cabba78-openshift-service-ca\") pod \"554e2752-4244-4f10-9c3b-21885cabba78\" (UID: \"554e2752-4244-4f10-9c3b-21885cabba78\") "
Jul 27 23:51:36.343924 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.343900    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554e2752-4244-4f10-9c3b-21885cabba78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "554e2752-4244-4f10-9c3b-21885cabba78" (UID: "554e2752-4244-4f10-9c3b-21885cabba78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:51:36.345997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.345976    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "554e2752-4244-4f10-9c3b-21885cabba78" (UID: "554e2752-4244-4f10-9c3b-21885cabba78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:51:36.346091 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.345992    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554e2752-4244-4f10-9c3b-21885cabba78-kube-api-access-fl4k8" (OuterVolumeSpecName: "kube-api-access-fl4k8") pod "554e2752-4244-4f10-9c3b-21885cabba78" (UID: "554e2752-4244-4f10-9c3b-21885cabba78"). InnerVolumeSpecName "kube-api-access-fl4k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:51:36.444696 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444663    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfvmv\" (UniqueName: \"kubernetes.io/projected/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-kube-api-access-pfvmv\") pod \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") "
Jul 27 23:51:36.444891 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444708    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-plugins-config-volume\") pod \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") "
Jul 27 23:51:36.444891 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444746    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-plugins-config-volume\") pod \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\" (UID: \"23ab7f79-bc0d-4598-8a07-3ea5788d57dc\") "
Jul 27 23:51:36.444891 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444761    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrwrl\" (UniqueName: \"kubernetes.io/projected/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-kube-api-access-lrwrl\") pod \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\" (UID: \"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278\") "
Jul 27 23:51:36.445059 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444978    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4k8\" (UniqueName: \"kubernetes.io/projected/554e2752-4244-4f10-9c3b-21885cabba78-kube-api-access-fl4k8\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:36.445059 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444990    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554e2752-4244-4f10-9c3b-21885cabba78-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:36.445059 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.444999    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554e2752-4244-4f10-9c3b-21885cabba78-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:36.445180 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.445076    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" (UID: "56ed64a1-3db5-4dc7-a2b4-6a14f43e2278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:51:36.445180 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.445160    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23ab7f79-bc0d-4598-8a07-3ea5788d57dc" (UID: "23ab7f79-bc0d-4598-8a07-3ea5788d57dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:51:36.446964 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.446943    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-kube-api-access-pfvmv" (OuterVolumeSpecName: "kube-api-access-pfvmv") pod "23ab7f79-bc0d-4598-8a07-3ea5788d57dc" (UID: "23ab7f79-bc0d-4598-8a07-3ea5788d57dc"). InnerVolumeSpecName "kube-api-access-pfvmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:51:36.447054 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.446969    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-kube-api-access-lrwrl" (OuterVolumeSpecName: "kube-api-access-lrwrl") pod "56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" (UID: "56ed64a1-3db5-4dc7-a2b4-6a14f43e2278"). InnerVolumeSpecName "kube-api-access-lrwrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:51:36.546504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.546420    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfvmv\" (UniqueName: \"kubernetes.io/projected/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-kube-api-access-pfvmv\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:36.546504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.546450    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:36.546504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.546460    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ab7f79-bc0d-4598-8a07-3ea5788d57dc-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:36.546504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:36.546469    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrwrl\" (UniqueName: \"kubernetes.io/projected/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278-kube-api-access-lrwrl\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:37.089892 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.089858    2570 generic.go:356] "Generic (PLEG): container finished" podID="56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" containerID="d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e" exitCode=0
Jul 27 23:51:37.090325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.089905    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"
Jul 27 23:51:37.090325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.089946    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz" event={"ID":"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278","Type":"ContainerDied","Data":"d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e"}
Jul 27 23:51:37.090325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.090009    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz" event={"ID":"56ed64a1-3db5-4dc7-a2b4-6a14f43e2278","Type":"ContainerDied","Data":"3bcc87de946e87cc251924fdc6de6eeab0f8cce4558f6c5cf5337e14b8e1670f"}
Jul 27 23:51:37.090325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.090034    2570 scope.go:117] "RemoveContainer" containerID="d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e"
Jul 27 23:51:37.091306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.091228    2570 generic.go:356] "Generic (PLEG): container finished" podID="23ab7f79-bc0d-4598-8a07-3ea5788d57dc" containerID="0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87" exitCode=0
Jul 27 23:51:37.091306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.091265    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"
Jul 27 23:51:37.091306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.091301    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6" event={"ID":"23ab7f79-bc0d-4598-8a07-3ea5788d57dc","Type":"ContainerDied","Data":"0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87"}
Jul 27 23:51:37.091523 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.091343    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6" event={"ID":"23ab7f79-bc0d-4598-8a07-3ea5788d57dc","Type":"ContainerDied","Data":"ce25eec35c0d98098b3feab768e63391ddb4f6408ff0ea8a95bdc5dfe8f5a3a8"}
Jul 27 23:51:37.092724 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.092698    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v" event={"ID":"554e2752-4244-4f10-9c3b-21885cabba78","Type":"ContainerDied","Data":"f4c9e65725d73c9890bb0bc7a2a5b2e6f4adbb03570d84afbfda2f65796c622f"}
Jul 27 23:51:37.092724 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.092723    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"
Jul 27 23:51:37.100343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.100317    2570 scope.go:117] "RemoveContainer" containerID="d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e"
Jul 27 23:51:37.100696 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:51:37.100674    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e\": container with ID starting with d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e not found: ID does not exist" containerID="d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e"
Jul 27 23:51:37.100805 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.100706    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e"} err="failed to get container status \"d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e\": rpc error: code = NotFound desc = could not find container \"d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e\": container with ID starting with d1c0c088cf09b7a2861e29dbfec1124cf668f55878a7d34966cbbd9a60dfff6e not found: ID does not exist"
Jul 27 23:51:37.100805 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.100726    2570 scope.go:117] "RemoveContainer" containerID="0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87"
Jul 27 23:51:37.109291 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.109277    2570 scope.go:117] "RemoveContainer" containerID="0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87"
Jul 27 23:51:37.109567 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:51:37.109549    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87\": container with ID starting with 0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87 not found: ID does not exist" containerID="0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87"
Jul 27 23:51:37.109623 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.109575    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87"} err="failed to get container status \"0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87\": rpc error: code = NotFound desc = could not find container \"0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87\": container with ID starting with 0c0fe7a8499ed5c18c40e669c8cc7dbc41a3760682a02fa352ca4d3b6af02f87 not found: ID does not exist"
Jul 27 23:51:37.109623 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.109591    2570 scope.go:117] "RemoveContainer" containerID="d2ad89cc4ba11ea668da0d6f804334b23e961900f7dcda790112f3601f3027bb"
Jul 27 23:51:37.117964 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.117938    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"]
Jul 27 23:51:37.119751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.119730    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0050c7fc-5c67795848-9fdx6"]
Jul 27 23:51:37.128975 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.128956    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"]
Jul 27 23:51:37.132527 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.132505    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0050c7fc-78c4d76897-s6h8v"]
Jul 27 23:51:37.142251 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.142229    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"]
Jul 27 23:51:37.144507 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.144487    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0050c7fc-5d54f9f798-bgmrz"]
Jul 27 23:51:37.521809 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.521772    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ab7f79-bc0d-4598-8a07-3ea5788d57dc" path="/var/lib/kubelet/pods/23ab7f79-bc0d-4598-8a07-3ea5788d57dc/volumes"
Jul 27 23:51:37.522167 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.522155    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554e2752-4244-4f10-9c3b-21885cabba78" path="/var/lib/kubelet/pods/554e2752-4244-4f10-9c3b-21885cabba78/volumes"
Jul 27 23:51:37.522481 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:37.522469    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" path="/var/lib/kubelet/pods/56ed64a1-3db5-4dc7-a2b4-6a14f43e2278/volumes"
Jul 27 23:51:46.864673 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:46.864637    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"]
Jul 27 23:51:46.865247 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:46.864896    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" containerID="cri-o://8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b" gracePeriod=30
Jul 27 23:51:49.143131 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:49.143085    2570 patch_prober.go:28] interesting pod/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 27 23:51:49.143531 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:49.143145    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 23:51:52.123776 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.123706    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:52.168503 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.168469    2570 generic.go:356] "Generic (PLEG): container finished" podID="69aec922-4d70-4c61-851a-f7d213f077e0" containerID="8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b" exitCode=0
Jul 27 23:51:52.168681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.168521    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"
Jul 27 23:51:52.168681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.168552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" event={"ID":"69aec922-4d70-4c61-851a-f7d213f077e0","Type":"ContainerDied","Data":"8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b"}
Jul 27 23:51:52.168681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.168603    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p" event={"ID":"69aec922-4d70-4c61-851a-f7d213f077e0","Type":"ContainerDied","Data":"de734943910951b5eedac4a9bfaeb06e58934044827c1d2d19b80a9269861d33"}
Jul 27 23:51:52.168681 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.168627    2570 scope.go:117] "RemoveContainer" containerID="8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b"
Jul 27 23:51:52.177680 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.177659    2570 scope.go:117] "RemoveContainer" containerID="fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"
Jul 27 23:51:52.185486 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.185463    2570 scope.go:117] "RemoveContainer" containerID="8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b"
Jul 27 23:51:52.185747 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:51:52.185727    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b\": container with ID starting with 8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b not found: ID does not exist" containerID="8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b"
Jul 27 23:51:52.185806 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.185758    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b"} err="failed to get container status \"8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b\": rpc error: code = NotFound desc = could not find container \"8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b\": container with ID starting with 8e8a12f73959fd4bbe7839a65552f97dbb886f7a6ae70e1b1727acf36f66eb3b not found: ID does not exist"
Jul 27 23:51:52.185806 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.185778    2570 scope.go:117] "RemoveContainer" containerID="fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"
Jul 27 23:51:52.186002 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:51:52.185982    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb\": container with ID starting with fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb not found: ID does not exist" containerID="fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"
Jul 27 23:51:52.186064 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.186014    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb"} err="failed to get container status \"fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb\": rpc error: code = NotFound desc = could not find container \"fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb\": container with ID starting with fb09540c2ceb3689d7e24a41d2547ddc1d29a46fd4d30bb2a67b35a152a534cb not found: ID does not exist"
Jul 27 23:51:52.189286 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189269    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-envoy\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189373 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189312    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-istio-token\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189373 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189330    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-data\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189373 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189346    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9n9k\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-kube-api-access-h9n9k\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189387    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69aec922-4d70-4c61-851a-f7d213f077e0-istiod-ca-cert\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189412    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-socket\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189438    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69aec922-4d70-4c61-851a-f7d213f077e0-istio-podinfo\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189534 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189514    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-certs\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189744 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189545    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-credential-socket\") pod \"69aec922-4d70-4c61-851a-f7d213f077e0\" (UID: \"69aec922-4d70-4c61-851a-f7d213f077e0\") "
Jul 27 23:51:52.189800 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189771    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69aec922-4d70-4c61-851a-f7d213f077e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:51:52.189892 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189873    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69aec922-4d70-4c61-851a-f7d213f077e0-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.189989 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.189948    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:51:52.190154 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.190130    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:51:52.190306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.190276    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:51:52.190445 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.190344    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:51:52.191536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.191514    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:51:52.191655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.191637    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-kube-api-access-h9n9k" (OuterVolumeSpecName: "kube-api-access-h9n9k") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "kube-api-access-h9n9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:51:52.192013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.191997    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:51:52.192111 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.192098    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69aec922-4d70-4c61-851a-f7d213f077e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69aec922-4d70-4c61-851a-f7d213f077e0" (UID: "69aec922-4d70-4c61-851a-f7d213f077e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:51:52.290790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290751    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69aec922-4d70-4c61-851a-f7d213f077e0-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.290790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290780    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.290790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290789    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.290790 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290799    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.291050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290811    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.291050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290818    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.291050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290827    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9n9k\" (UniqueName: \"kubernetes.io/projected/69aec922-4d70-4c61-851a-f7d213f077e0-kube-api-access-h9n9k\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.291050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.290835    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69aec922-4d70-4c61-851a-f7d213f077e0-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:51:52.491106 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.491067    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"]
Jul 27 23:51:52.495073 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:52.495040    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0050c7fc-openshift-default-578b4b9cf6-kml6p"]
Jul 27 23:51:53.521809 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:51:53.521777    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" path="/var/lib/kubelet/pods/69aec922-4d70-4c61-851a-f7d213f077e0/volumes"
Jul 27 23:52:05.913684 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.913605    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"]
Jul 27 23:52:05.914294 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914263    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" containerName="payload-processing"
Jul 27 23:52:05.914391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914287    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" containerName="payload-processing"
Jul 27 23:52:05.914391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914311    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy"
Jul 27 23:52:05.914391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914320    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy"
Jul 27 23:52:05.914391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914337    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy"
Jul 27 23:52:05.914391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914345    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914396    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ab7f79-bc0d-4598-8a07-3ea5788d57dc" containerName="payload-pre-processing"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914403    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ab7f79-bc0d-4598-8a07-3ea5788d57dc" containerName="payload-pre-processing"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914409    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554e2752-4244-4f10-9c3b-21885cabba78" containerName="maas-api"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914413    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="554e2752-4244-4f10-9c3b-21885cabba78" containerName="maas-api"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914504    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914515    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="554e2752-4244-4f10-9c3b-21885cabba78" containerName="maas-api"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914526    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ab7f79-bc0d-4598-8a07-3ea5788d57dc" containerName="payload-pre-processing"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914536    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="69aec922-4d70-4c61-851a-f7d213f077e0" containerName="istio-proxy"
Jul 27 23:52:05.914625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.914542    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ed64a1-3db5-4dc7-a2b4-6a14f43e2278" containerName="payload-processing"
Jul 27 23:52:05.917769 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.917749    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:05.920608 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.920589    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27ab6b49-openshift-default-dockercfg-zd99h\""
Jul 27 23:52:05.930463 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:05.930440    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"]
Jul 27 23:52:06.018760 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018729    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-envoy\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.018956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-credential-socket\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.018956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-socket\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.018956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzhjq\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-kube-api-access-lzhjq\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.018956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-data\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.018956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018920    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-token\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.018956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.018955    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-certs\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.019182 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.019037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istiod-ca-cert\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.019182 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.019122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-podinfo\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.119972 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.119937    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-socket\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.119982    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzhjq\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-kube-api-access-lzhjq\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120012    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-data\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-token\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-certs\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istiod-ca-cert\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120171 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-podinfo\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-envoy\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120408    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-credential-socket\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120418    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-socket\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120501    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-certs\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120743    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-credential-socket\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.120766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120746    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-data\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.121120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.120899    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istiod-ca-cert\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.122645 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.122618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-envoy\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.122958 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.122938    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-podinfo\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.130940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.130921    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-token\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.131473 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.131449    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzhjq\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-kube-api-access-lzhjq\") pod \"e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") " pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.229247 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.229147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:06.373942 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.373891    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"]
Jul 27 23:52:06.376855 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:52:06.376831    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c24c984_37e9_40d7_afdb_0153c43f3c9d.slice/crio-919b7d9edc3d6eee609266636dbfdd0febb2c3391f31a7660c8d4b1fc9a293ed WatchSource:0}: Error finding container 919b7d9edc3d6eee609266636dbfdd0febb2c3391f31a7660c8d4b1fc9a293ed: Status 404 returned error can't find the container with id 919b7d9edc3d6eee609266636dbfdd0febb2c3391f31a7660c8d4b1fc9a293ed
Jul 27 23:52:06.378865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.378827    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:52:06.378970 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.378916    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:52:06.378970 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:06.378960    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:52:07.242705 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:07.242665    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" event={"ID":"0c24c984-37e9-40d7-afdb-0153c43f3c9d","Type":"ContainerStarted","Data":"7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"}
Jul 27 23:52:07.243114 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:07.242726    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" event={"ID":"0c24c984-37e9-40d7-afdb-0153c43f3c9d","Type":"ContainerStarted","Data":"919b7d9edc3d6eee609266636dbfdd0febb2c3391f31a7660c8d4b1fc9a293ed"}
Jul 27 23:52:07.263266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:07.263215    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" podStartSLOduration=2.263197633 podStartE2EDuration="2.263197633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:52:07.262033649 +0000 UTC m=+2152.213377055" watchObservedRunningTime="2026-07-27 23:52:07.263197633 +0000 UTC m=+2152.214541021"
Jul 27 23:52:08.229391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:08.229332    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:08.234529 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:08.234504    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:08.249134 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:08.249109    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:08.250029 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:08.250013    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:09.210507 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.210471    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"]
Jul 27 23:52:09.214259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.214237    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.216912 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.216891    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-27ab6b49-serving-cert\""
Jul 27 23:52:09.224924 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.224896    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"]
Jul 27 23:52:09.248324 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.248289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18dce08-13e1-448f-b064-98c64f6756ad-openshift-service-ca\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.248688 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.248665    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtqt\" (UniqueName: \"kubernetes.io/projected/e18dce08-13e1-448f-b064-98c64f6756ad-kube-api-access-5mtqt\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.248734 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.248706    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18dce08-13e1-448f-b064-98c64f6756ad-maas-api-tls\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.349316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.349274    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18dce08-13e1-448f-b064-98c64f6756ad-openshift-service-ca\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.349832 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.349467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtqt\" (UniqueName: \"kubernetes.io/projected/e18dce08-13e1-448f-b064-98c64f6756ad-kube-api-access-5mtqt\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.349832 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.349505    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18dce08-13e1-448f-b064-98c64f6756ad-maas-api-tls\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.349978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.349956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18dce08-13e1-448f-b064-98c64f6756ad-openshift-service-ca\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.352062 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.352039    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18dce08-13e1-448f-b064-98c64f6756ad-maas-api-tls\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.357602 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.357578    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtqt\" (UniqueName: \"kubernetes.io/projected/e18dce08-13e1-448f-b064-98c64f6756ad-kube-api-access-5mtqt\") pod \"maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.526160 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.526077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:09.670127 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.670103    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"]
Jul 27 23:52:09.672069 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:52:09.672043    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18dce08_13e1_448f_b064_98c64f6756ad.slice/crio-50476ce6dd0f28b895535640222ccc0d3e3ac1ebea212f3ebff0e2c9ef3f6c42 WatchSource:0}: Error finding container 50476ce6dd0f28b895535640222ccc0d3e3ac1ebea212f3ebff0e2c9ef3f6c42: Status 404 returned error can't find the container with id 50476ce6dd0f28b895535640222ccc0d3e3ac1ebea212f3ebff0e2c9ef3f6c42
Jul 27 23:52:09.849821 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.849733    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"]
Jul 27 23:52:09.854462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.854439    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:09.857000 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.856980    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-27ab6b49-dockercfg-jmp8s\""
Jul 27 23:52:09.857126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.856983    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-27ab6b49\""
Jul 27 23:52:09.863147 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.863121    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"]
Jul 27 23:52:09.893558 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.893522    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"]
Jul 27 23:52:09.897020 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.896996    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:09.907328 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.907297    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"]
Jul 27 23:52:09.954216 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.954162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxwq\" (UniqueName: \"kubernetes.io/projected/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-kube-api-access-4lxwq\") pod \"payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") " pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:09.954460 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.954230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2hss\" (UniqueName: \"kubernetes.io/projected/acfdd848-11ab-467d-9b86-3e072648973c-kube-api-access-j2hss\") pod \"payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:09.954460 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.954279    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdd848-11ab-467d-9b86-3e072648973c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:09.954460 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:09.954414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") " pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:10.054895 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.054873    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdd848-11ab-467d-9b86-3e072648973c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:10.055060 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.055025    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") " pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:10.055129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.055102    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxwq\" (UniqueName: \"kubernetes.io/projected/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-kube-api-access-4lxwq\") pod \"payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") " pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:10.055185 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.055152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2hss\" (UniqueName: \"kubernetes.io/projected/acfdd848-11ab-467d-9b86-3e072648973c-kube-api-access-j2hss\") pod \"payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:10.055773 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.055747    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdd848-11ab-467d-9b86-3e072648973c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:10.055978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.055956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") " pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:10.064820 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.064798    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxwq\" (UniqueName: \"kubernetes.io/projected/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-kube-api-access-4lxwq\") pod \"payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") " pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:10.065758 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.065730    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2hss\" (UniqueName: \"kubernetes.io/projected/acfdd848-11ab-467d-9b86-3e072648973c-kube-api-access-j2hss\") pod \"payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:10.166464 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.166423    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:10.209084 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.209045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:10.267206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.266825    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs" event={"ID":"e18dce08-13e1-448f-b064-98c64f6756ad","Type":"ContainerStarted","Data":"f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9"}
Jul 27 23:52:10.267206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.266897    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs" event={"ID":"e18dce08-13e1-448f-b064-98c64f6756ad","Type":"ContainerStarted","Data":"50476ce6dd0f28b895535640222ccc0d3e3ac1ebea212f3ebff0e2c9ef3f6c42"}
Jul 27 23:52:10.288641 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.288573    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs" podStartSLOduration=0.911761091 podStartE2EDuration="1.288548728s" totalImagesPullingTime="376.787637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:52:10.282990793 +0000 UTC m=+2155.234334181" watchObservedRunningTime="2026-07-27 23:52:10.288548728 +0000 UTC m=+2155.239892114"
Jul 27 23:52:10.327849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.327816    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"]
Jul 27 23:52:10.331272 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:52:10.331194    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfdd848_11ab_467d_9b86_3e072648973c.slice/crio-d0194981d9ead530e209443e02dbbaa9533c5aa1388a3da4bd7b46d474be1171 WatchSource:0}: Error finding container d0194981d9ead530e209443e02dbbaa9533c5aa1388a3da4bd7b46d474be1171: Status 404 returned error can't find the container with id d0194981d9ead530e209443e02dbbaa9533c5aa1388a3da4bd7b46d474be1171
Jul 27 23:52:10.388822 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:10.388770    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"]
Jul 27 23:52:10.390556 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:52:10.390516    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ec92c9_9baa_4d7c_9b5b_5292f80577d3.slice/crio-7bfeceeba12cdf466c5c8d37572d12f7b8b2d71682f2b6708c99cfa200123322 WatchSource:0}: Error finding container 7bfeceeba12cdf466c5c8d37572d12f7b8b2d71682f2b6708c99cfa200123322: Status 404 returned error can't find the container with id 7bfeceeba12cdf466c5c8d37572d12f7b8b2d71682f2b6708c99cfa200123322
Jul 27 23:52:11.273978 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.273931    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm" event={"ID":"42ec92c9-9baa-4d7c-9b5b-5292f80577d3","Type":"ContainerStarted","Data":"d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682"}
Jul 27 23:52:11.274186 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.274000    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:11.274186 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.274025    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm" event={"ID":"42ec92c9-9baa-4d7c-9b5b-5292f80577d3","Type":"ContainerStarted","Data":"7bfeceeba12cdf466c5c8d37572d12f7b8b2d71682f2b6708c99cfa200123322"}
Jul 27 23:52:11.275584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.275553    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8" event={"ID":"acfdd848-11ab-467d-9b86-3e072648973c","Type":"ContainerStarted","Data":"09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9"}
Jul 27 23:52:11.275727 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.275610    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8" event={"ID":"acfdd848-11ab-467d-9b86-3e072648973c","Type":"ContainerStarted","Data":"d0194981d9ead530e209443e02dbbaa9533c5aa1388a3da4bd7b46d474be1171"}
Jul 27 23:52:11.275727 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.275672    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:11.275727 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.275715    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:11.289405 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.289324    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm" podStartSLOduration=2.289305026 podStartE2EDuration="2.289305026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:52:11.289216035 +0000 UTC m=+2156.240559448" watchObservedRunningTime="2026-07-27 23:52:11.289305026 +0000 UTC m=+2156.240648433"
Jul 27 23:52:11.304591 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:11.304547    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8" podStartSLOduration=2.3045331190000002 podStartE2EDuration="2.304533119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:52:11.30357139 +0000 UTC m=+2156.254914806" watchObservedRunningTime="2026-07-27 23:52:11.304533119 +0000 UTC m=+2156.255876505"
Jul 27 23:52:17.286945 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:17.286915    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:52:22.282997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.282964    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:52:22.283475 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.283073    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:52:22.336663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.336625    2570 generic.go:356] "Generic (PLEG): container finished" podID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerID="7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd" exitCode=0
Jul 27 23:52:22.336875 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.336686    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" event={"ID":"0c24c984-37e9-40d7-afdb-0153c43f3c9d","Type":"ContainerDied","Data":"7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"}
Jul 27 23:52:22.337303 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.337282    2570 scope.go:117] "RemoveContainer" containerID="7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"
Jul 27 23:52:22.338533 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.338501    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:52:22.338623 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.338604    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:52:22.338686 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:22.338662    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:52:23.250009 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:23.249967    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:23.343874 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:23.343836    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" event={"ID":"0c24c984-37e9-40d7-afdb-0153c43f3c9d","Type":"ContainerStarted","Data":"172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842"}
Jul 27 23:52:24.230258 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:24.230221    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:24.231487 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:24.231460    2570 patch_prober.go:28] interesting pod/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 23:52:24.231621 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:24.231519    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 23:52:25.229980 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:25.229936    2570 patch_prober.go:28] interesting pod/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 23:52:25.230526 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:25.230009    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 23:52:26.229692 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:26.229654    2570 patch_prober.go:28] interesting pod/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 27 23:52:26.229873 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:26.229714    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 27 23:52:27.236100 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:27.236054    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:27.236800 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:27.236776    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:52:27.237652 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:52:27.237633    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:53:09.189001 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.188951    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"]
Jul 27 23:53:09.189638 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.189230    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs" podUID="e18dce08-13e1-448f-b064-98c64f6756ad" containerName="maas-api" containerID="cri-o://f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9" gracePeriod=30
Jul 27 23:53:09.240460 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.240419    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"]
Jul 27 23:53:09.240766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.240708    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm" podUID="42ec92c9-9baa-4d7c-9b5b-5292f80577d3" containerName="payload-processing" containerID="cri-o://d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682" gracePeriod=30
Jul 27 23:53:09.245747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.244929    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"]
Jul 27 23:53:09.245747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.245145    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8" podUID="acfdd848-11ab-467d-9b86-3e072648973c" containerName="payload-pre-processing" containerID="cri-o://09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9" gracePeriod=30
Jul 27 23:53:09.469151 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.469125    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:53:09.512516 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.512479    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18dce08-13e1-448f-b064-98c64f6756ad-openshift-service-ca\") pod \"e18dce08-13e1-448f-b064-98c64f6756ad\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") "
Jul 27 23:53:09.512516 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.512516    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mtqt\" (UniqueName: \"kubernetes.io/projected/e18dce08-13e1-448f-b064-98c64f6756ad-kube-api-access-5mtqt\") pod \"e18dce08-13e1-448f-b064-98c64f6756ad\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") "
Jul 27 23:53:09.512766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.512553    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18dce08-13e1-448f-b064-98c64f6756ad-maas-api-tls\") pod \"e18dce08-13e1-448f-b064-98c64f6756ad\" (UID: \"e18dce08-13e1-448f-b064-98c64f6756ad\") "
Jul 27 23:53:09.512914 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.512884    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18dce08-13e1-448f-b064-98c64f6756ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e18dce08-13e1-448f-b064-98c64f6756ad" (UID: "e18dce08-13e1-448f-b064-98c64f6756ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:53:09.516575 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.516503    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18dce08-13e1-448f-b064-98c64f6756ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e18dce08-13e1-448f-b064-98c64f6756ad" (UID: "e18dce08-13e1-448f-b064-98c64f6756ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:53:09.516575 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.516552    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18dce08-13e1-448f-b064-98c64f6756ad-kube-api-access-5mtqt" (OuterVolumeSpecName: "kube-api-access-5mtqt") pod "e18dce08-13e1-448f-b064-98c64f6756ad" (UID: "e18dce08-13e1-448f-b064-98c64f6756ad"). InnerVolumeSpecName "kube-api-access-5mtqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:53:09.538837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.538806    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:53:09.546762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.546741    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:53:09.579409 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.579379    2570 generic.go:356] "Generic (PLEG): container finished" podID="e18dce08-13e1-448f-b064-98c64f6756ad" containerID="f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9" exitCode=0
Jul 27 23:53:09.579568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.579436    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"
Jul 27 23:53:09.579568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.579444    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs" event={"ID":"e18dce08-13e1-448f-b064-98c64f6756ad","Type":"ContainerDied","Data":"f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9"}
Jul 27 23:53:09.579568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.579500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs" event={"ID":"e18dce08-13e1-448f-b064-98c64f6756ad","Type":"ContainerDied","Data":"50476ce6dd0f28b895535640222ccc0d3e3ac1ebea212f3ebff0e2c9ef3f6c42"}
Jul 27 23:53:09.579568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.579527    2570 scope.go:117] "RemoveContainer" containerID="f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9"
Jul 27 23:53:09.580892 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.580833    2570 generic.go:356] "Generic (PLEG): container finished" podID="42ec92c9-9baa-4d7c-9b5b-5292f80577d3" containerID="d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682" exitCode=0
Jul 27 23:53:09.580892 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.580865    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm" event={"ID":"42ec92c9-9baa-4d7c-9b5b-5292f80577d3","Type":"ContainerDied","Data":"d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682"}
Jul 27 23:53:09.580892 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.580882    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"
Jul 27 23:53:09.581045 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.580910    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm" event={"ID":"42ec92c9-9baa-4d7c-9b5b-5292f80577d3","Type":"ContainerDied","Data":"7bfeceeba12cdf466c5c8d37572d12f7b8b2d71682f2b6708c99cfa200123322"}
Jul 27 23:53:09.582672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.582650    2570 generic.go:356] "Generic (PLEG): container finished" podID="acfdd848-11ab-467d-9b86-3e072648973c" containerID="09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9" exitCode=0
Jul 27 23:53:09.582770 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.582683    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"
Jul 27 23:53:09.582770 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.582708    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8" event={"ID":"acfdd848-11ab-467d-9b86-3e072648973c","Type":"ContainerDied","Data":"09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9"}
Jul 27 23:53:09.582770 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.582744    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8" event={"ID":"acfdd848-11ab-467d-9b86-3e072648973c","Type":"ContainerDied","Data":"d0194981d9ead530e209443e02dbbaa9533c5aa1388a3da4bd7b46d474be1171"}
Jul 27 23:53:09.590633 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.590616    2570 scope.go:117] "RemoveContainer" containerID="f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9"
Jul 27 23:53:09.590906 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:53:09.590881    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9\": container with ID starting with f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9 not found: ID does not exist" containerID="f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9"
Jul 27 23:53:09.590999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.590914    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9"} err="failed to get container status \"f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9\": rpc error: code = NotFound desc = could not find container \"f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9\": container with ID starting with f8af27ccde52300ea0cff6cac8a36d71ccf80ae788ce0efb4cb4da30a8602df9 not found: ID does not exist"
Jul 27 23:53:09.590999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.590931    2570 scope.go:117] "RemoveContainer" containerID="d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682"
Jul 27 23:53:09.596371 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.596329    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"]
Jul 27 23:53:09.599723 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.599704    2570 scope.go:117] "RemoveContainer" containerID="d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682"
Jul 27 23:53:09.600025 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:53:09.600006    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682\": container with ID starting with d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682 not found: ID does not exist" containerID="d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682"
Jul 27 23:53:09.600078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.600033    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682"} err="failed to get container status \"d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682\": rpc error: code = NotFound desc = could not find container \"d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682\": container with ID starting with d8c9a8bfc6a4ad04bfb73291c3d501bbacad703bd27a3b21eec01be4bb28c682 not found: ID does not exist"
Jul 27 23:53:09.600078 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.600053    2570 scope.go:117] "RemoveContainer" containerID="09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9"
Jul 27 23:53:09.600173 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.600083    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27ab6b49-85b6777594-9wvgs"]
Jul 27 23:53:09.608343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.608326    2570 scope.go:117] "RemoveContainer" containerID="09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9"
Jul 27 23:53:09.608718 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:53:09.608698    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9\": container with ID starting with 09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9 not found: ID does not exist" containerID="09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9"
Jul 27 23:53:09.608775 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.608723    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9"} err="failed to get container status \"09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9\": rpc error: code = NotFound desc = could not find container \"09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9\": container with ID starting with 09fa0b53268d7198c9d631c183d1e8c475c776af27e87b0da208587bc90568f9 not found: ID does not exist"
Jul 27 23:53:09.613346 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.613330    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdd848-11ab-467d-9b86-3e072648973c-plugins-config-volume\") pod \"acfdd848-11ab-467d-9b86-3e072648973c\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") "
Jul 27 23:53:09.613434 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.613384    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-plugins-config-volume\") pod \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") "
Jul 27 23:53:09.613434 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.613427    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2hss\" (UniqueName: \"kubernetes.io/projected/acfdd848-11ab-467d-9b86-3e072648973c-kube-api-access-j2hss\") pod \"acfdd848-11ab-467d-9b86-3e072648973c\" (UID: \"acfdd848-11ab-467d-9b86-3e072648973c\") "
Jul 27 23:53:09.613515 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.613504    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lxwq\" (UniqueName: \"kubernetes.io/projected/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-kube-api-access-4lxwq\") pod \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\" (UID: \"42ec92c9-9baa-4d7c-9b5b-5292f80577d3\") "
Jul 27 23:53:09.613726 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.613687    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfdd848-11ab-467d-9b86-3e072648973c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acfdd848-11ab-467d-9b86-3e072648973c" (UID: "acfdd848-11ab-467d-9b86-3e072648973c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:53:09.613842 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.613718    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42ec92c9-9baa-4d7c-9b5b-5292f80577d3" (UID: "42ec92c9-9baa-4d7c-9b5b-5292f80577d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:53:09.614229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.614103    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18dce08-13e1-448f-b064-98c64f6756ad-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.614229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.614135    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mtqt\" (UniqueName: \"kubernetes.io/projected/e18dce08-13e1-448f-b064-98c64f6756ad-kube-api-access-5mtqt\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.614229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.614163    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfdd848-11ab-467d-9b86-3e072648973c-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.614229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.614176    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.614229 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.614190    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18dce08-13e1-448f-b064-98c64f6756ad-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.615747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.615725    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfdd848-11ab-467d-9b86-3e072648973c-kube-api-access-j2hss" (OuterVolumeSpecName: "kube-api-access-j2hss") pod "acfdd848-11ab-467d-9b86-3e072648973c" (UID: "acfdd848-11ab-467d-9b86-3e072648973c"). InnerVolumeSpecName "kube-api-access-j2hss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:53:09.615831 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.615764    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-kube-api-access-4lxwq" (OuterVolumeSpecName: "kube-api-access-4lxwq") pod "42ec92c9-9baa-4d7c-9b5b-5292f80577d3" (UID: "42ec92c9-9baa-4d7c-9b5b-5292f80577d3"). InnerVolumeSpecName "kube-api-access-4lxwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:53:09.715468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.715340    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lxwq\" (UniqueName: \"kubernetes.io/projected/42ec92c9-9baa-4d7c-9b5b-5292f80577d3-kube-api-access-4lxwq\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.715468 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.715399    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2hss\" (UniqueName: \"kubernetes.io/projected/acfdd848-11ab-467d-9b86-3e072648973c-kube-api-access-j2hss\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:53:09.903658 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.903624    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"]
Jul 27 23:53:09.907456 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.907426    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-27ab6b49-595fbd94bb-fjcqm"]
Jul 27 23:53:09.916922 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.916887    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"]
Jul 27 23:53:09.920383 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:09.920346    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-27ab6b49-66d8ffd456-gcpm8"]
Jul 27 23:53:11.523292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:11.523252    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ec92c9-9baa-4d7c-9b5b-5292f80577d3" path="/var/lib/kubelet/pods/42ec92c9-9baa-4d7c-9b5b-5292f80577d3/volumes"
Jul 27 23:53:11.523812 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:11.523794    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acfdd848-11ab-467d-9b86-3e072648973c" path="/var/lib/kubelet/pods/acfdd848-11ab-467d-9b86-3e072648973c/volumes"
Jul 27 23:53:11.524205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:11.524184    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e18dce08-13e1-448f-b064-98c64f6756ad" path="/var/lib/kubelet/pods/e18dce08-13e1-448f-b064-98c64f6756ad/volumes"
Jul 27 23:53:15.701591 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:53:15.701555    2570 scope.go:117] "RemoveContainer" containerID="a801e04eb93b09866c3204afaa97b054a1841ecebdd2d033e84fc7c86dade77f"
Jul 27 23:54:04.493070 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:04.493034    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"]
Jul 27 23:54:04.493667 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:04.493291    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy" containerID="cri-o://172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842" gracePeriod=30
Jul 27 23:54:09.744400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.744375    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:54:09.893945 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.893914    2570 generic.go:356] "Generic (PLEG): container finished" podID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerID="172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842" exitCode=0
Jul 27 23:54:09.894124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.893964    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"
Jul 27 23:54:09.894124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.893996    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" event={"ID":"0c24c984-37e9-40d7-afdb-0153c43f3c9d","Type":"ContainerDied","Data":"172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842"}
Jul 27 23:54:09.894124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.894049    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql" event={"ID":"0c24c984-37e9-40d7-afdb-0153c43f3c9d","Type":"ContainerDied","Data":"919b7d9edc3d6eee609266636dbfdd0febb2c3391f31a7660c8d4b1fc9a293ed"}
Jul 27 23:54:09.894124 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.894072    2570 scope.go:117] "RemoveContainer" containerID="172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842"
Jul 27 23:54:09.896269 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896249    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzhjq\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-kube-api-access-lzhjq\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896404 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896276    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-podinfo\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896404 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896346    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-data\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896404 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896398    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-envoy\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896414    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-credential-socket\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896438    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-certs\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896476    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-socket\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896496    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istiod-ca-cert\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896518    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-token\") pod \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\" (UID: \"0c24c984-37e9-40d7-afdb-0153c43f3c9d\") "
Jul 27 23:54:09.896847 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896734    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:54:09.896847 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896826    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:54:09.896949 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.896851    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.897160 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.897127    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:54:09.897586 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.897303    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:54:09.897586 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.897401    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:54:09.898963 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.898940    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:54:09.899081 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.899063    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-kube-api-access-lzhjq" (OuterVolumeSpecName: "kube-api-access-lzhjq") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "kube-api-access-lzhjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:54:09.899213 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.899189    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:54:09.899387 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.899342    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c24c984-37e9-40d7-afdb-0153c43f3c9d" (UID: "0c24c984-37e9-40d7-afdb-0153c43f3c9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:54:09.916426 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.916405    2570 scope.go:117] "RemoveContainer" containerID="7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"
Jul 27 23:54:09.924266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.924242    2570 scope.go:117] "RemoveContainer" containerID="172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842"
Jul 27 23:54:09.924536 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:54:09.924519    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842\": container with ID starting with 172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842 not found: ID does not exist" containerID="172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842"
Jul 27 23:54:09.924601 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.924551    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842"} err="failed to get container status \"172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842\": rpc error: code = NotFound desc = could not find container \"172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842\": container with ID starting with 172bbaf61d0e91e1d027c1983753203249e4650429fa6387a33304d160b1d842 not found: ID does not exist"
Jul 27 23:54:09.924601 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.924572    2570 scope.go:117] "RemoveContainer" containerID="7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"
Jul 27 23:54:09.924784 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:54:09.924766    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd\": container with ID starting with 7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd not found: ID does not exist" containerID="7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"
Jul 27 23:54:09.924823 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.924791    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd"} err="failed to get container status \"7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd\": rpc error: code = NotFound desc = could not find container \"7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd\": container with ID starting with 7ff5372150237eb73f11150f0b01d43499d3e6a5ba1c1aaa8f462ee668036fcd not found: ID does not exist"
Jul 27 23:54:09.997994 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.997957    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzhjq\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-kube-api-access-lzhjq\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.997994 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.997989    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.997994 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.998001    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.998210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.998011    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.998210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.998019    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.998210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.998026    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c24c984-37e9-40d7-afdb-0153c43f3c9d-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.998210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.998034    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:09.998210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:09.998041    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c24c984-37e9-40d7-afdb-0153c43f3c9d-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:54:10.216737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:10.216703    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"]
Jul 27 23:54:10.220423 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:10.220400    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27ab6b49-openshift-default-57ff9b6f48-xcpql"]
Jul 27 23:54:11.522069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:11.522036    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" path="/var/lib/kubelet/pods/0c24c984-37e9-40d7-afdb-0153c43f3c9d/volumes"
Jul 27 23:54:59.293345 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293309    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"]
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293701    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293713    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293727    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293738    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293757    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e18dce08-13e1-448f-b064-98c64f6756ad" containerName="maas-api"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293763    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e18dce08-13e1-448f-b064-98c64f6756ad" containerName="maas-api"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293794    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acfdd848-11ab-467d-9b86-3e072648973c" containerName="payload-pre-processing"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293801    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="acfdd848-11ab-467d-9b86-3e072648973c" containerName="payload-pre-processing"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293816    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ec92c9-9baa-4d7c-9b5b-5292f80577d3" containerName="payload-processing"
Jul 27 23:54:59.293851 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293826    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ec92c9-9baa-4d7c-9b5b-5292f80577d3" containerName="payload-processing"
Jul 27 23:54:59.294156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293880    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy"
Jul 27 23:54:59.294156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293891    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="acfdd848-11ab-467d-9b86-3e072648973c" containerName="payload-pre-processing"
Jul 27 23:54:59.294156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293903    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e18dce08-13e1-448f-b064-98c64f6756ad" containerName="maas-api"
Jul 27 23:54:59.294156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.293909    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ec92c9-9baa-4d7c-9b5b-5292f80577d3" containerName="payload-processing"
Jul 27 23:54:59.296701 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.296685    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.299325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.299304    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-05a1fc-openshift-default-dockercfg-qkcp9\""
Jul 27 23:54:59.310091 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.309653    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"]
Jul 27 23:54:59.341498 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341463    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-token\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341498 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2211b-a134-4921-b89c-8dfecdb8c865-istiod-ca-cert\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341547    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-envoy\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341586    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-podinfo\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341621    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mt8\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-kube-api-access-68mt8\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-certs\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341688    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-socket\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341707    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-data\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.341970 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.341765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-credential-socket\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443053 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443013    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-envoy\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443244 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443057    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-podinfo\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443244 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mt8\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-kube-api-access-68mt8\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443244 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-certs\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443244 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443144    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-socket\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443521 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443371    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-data\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443521 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-credential-socket\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443637 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443563    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-token\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443637 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2211b-a134-4921-b89c-8dfecdb8c865-istiod-ca-cert\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443637 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443621    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-certs\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443683    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-data\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443592    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-socket\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.443921 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.443876    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-credential-socket\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.444508 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.444484    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2211b-a134-4921-b89c-8dfecdb8c865-istiod-ca-cert\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.445704 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.445682    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-envoy\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.445992 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.445974    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-podinfo\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.451536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.451517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-token\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.451701 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.451681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mt8\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-kube-api-access-68mt8\") pod \"e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") " pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.615571 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.615482    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:54:59.754993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.754954    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"]
Jul 27 23:54:59.757024 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:54:59.756993    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d2211b_a134_4921_b89c_8dfecdb8c865.slice/crio-578752d4c47d09a681a17168850bc669d82f1719079f94f8d0e11f4ed2c8b8ad WatchSource:0}: Error finding container 578752d4c47d09a681a17168850bc669d82f1719079f94f8d0e11f4ed2c8b8ad: Status 404 returned error can't find the container with id 578752d4c47d09a681a17168850bc669d82f1719079f94f8d0e11f4ed2c8b8ad
Jul 27 23:54:59.759126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.759092    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:54:59.759225 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.759167    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:54:59.759225 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:54:59.759211    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:55:00.140970 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:00.140926    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" event={"ID":"58d2211b-a134-4921-b89c-8dfecdb8c865","Type":"ContainerStarted","Data":"c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"}
Jul 27 23:55:00.141152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:00.140997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" event={"ID":"58d2211b-a134-4921-b89c-8dfecdb8c865","Type":"ContainerStarted","Data":"578752d4c47d09a681a17168850bc669d82f1719079f94f8d0e11f4ed2c8b8ad"}
Jul 27 23:55:00.162141 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:00.162068    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" podStartSLOduration=1.162047815 podStartE2EDuration="1.162047815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:00.161545384 +0000 UTC m=+2325.112888822" watchObservedRunningTime="2026-07-27 23:55:00.162047815 +0000 UTC m=+2325.113391207"
Jul 27 23:55:00.616102 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:00.616005    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:00.621284 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:00.621262    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:01.146302 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:01.146274    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:01.147456 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:01.147432    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:02.229096 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.229065    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"]
Jul 27 23:55:02.229583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.229565    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c24c984-37e9-40d7-afdb-0153c43f3c9d" containerName="istio-proxy"
Jul 27 23:55:02.232805 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.232787    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.235280 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.235264    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-05a1fc-serving-cert\""
Jul 27 23:55:02.242745 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.242719    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"]
Jul 27 23:55:02.270109 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.270068    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.270300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.270160    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd5a78b-4362-4dbf-8159-875c6b827c25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.270300 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.270204    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr87p\" (UniqueName: \"kubernetes.io/projected/6cd5a78b-4362-4dbf-8159-875c6b827c25-kube-api-access-hr87p\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.370646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.370606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd5a78b-4362-4dbf-8159-875c6b827c25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.370646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.370651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr87p\" (UniqueName: \"kubernetes.io/projected/6cd5a78b-4362-4dbf-8159-875c6b827c25-kube-api-access-hr87p\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.370855 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.370719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.370855 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:55:02.370815    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-serving-cert: secret "maas-api-e2e-inf-a-05a1fc-serving-cert" not found
Jul 27 23:55:02.370923 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:55:02.370871    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls podName:6cd5a78b-4362-4dbf-8159-875c6b827c25 nodeName:}" failed. No retries permitted until 2026-07-27 23:55:02.870854396 +0000 UTC m=+2327.822197761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls") pod "maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" (UID: "6cd5a78b-4362-4dbf-8159-875c6b827c25") : secret "maas-api-e2e-inf-a-05a1fc-serving-cert" not found
Jul 27 23:55:02.371239 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.371219    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd5a78b-4362-4dbf-8159-875c6b827c25-openshift-service-ca\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.379425 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.379402    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr87p\" (UniqueName: \"kubernetes.io/projected/6cd5a78b-4362-4dbf-8159-875c6b827c25-kube-api-access-hr87p\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.858315 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.858275    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"]
Jul 27 23:55:02.861926 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.861908    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:02.865102 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.865077    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-05a1fc-dockercfg-z88t2\""
Jul 27 23:55:02.865245 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.865077    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-05a1fc\""
Jul 27 23:55:02.871536 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.871511    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"]
Jul 27 23:55:02.875209 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.875188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.877808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.877784    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls\") pod \"maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:02.882044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.882015    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"]
Jul 27 23:55:02.888033 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.888004    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:02.896516 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.896486    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"]
Jul 27 23:55:02.977377 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.977322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmdc\" (UniqueName: \"kubernetes.io/projected/ede34508-011f-41af-8f7e-35332581d005-kube-api-access-mcmdc\") pod \"payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:02.977573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.977472    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:02.977573 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.977513    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nt2r\" (UniqueName: \"kubernetes.io/projected/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-kube-api-access-7nt2r\") pod \"payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:02.977689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:02.977643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede34508-011f-41af-8f7e-35332581d005-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:03.078293 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.078248    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede34508-011f-41af-8f7e-35332581d005-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:03.078530 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.078306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmdc\" (UniqueName: \"kubernetes.io/projected/ede34508-011f-41af-8f7e-35332581d005-kube-api-access-mcmdc\") pod \"payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:03.078530 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.078392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:03.078530 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.078418    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nt2r\" (UniqueName: \"kubernetes.io/projected/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-kube-api-access-7nt2r\") pod \"payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:03.079062 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.079041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede34508-011f-41af-8f7e-35332581d005-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:03.079197 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.079178    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:03.087137 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.087109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmdc\" (UniqueName: \"kubernetes.io/projected/ede34508-011f-41af-8f7e-35332581d005-kube-api-access-mcmdc\") pod \"payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") " pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:03.087221 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.087154    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nt2r\" (UniqueName: \"kubernetes.io/projected/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-kube-api-access-7nt2r\") pod \"payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:03.145974 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.145941    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:03.174220 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.174191    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:03.201142 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.201092    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:03.312955 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.312899    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"]
Jul 27 23:55:03.315319 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:55:03.315290    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd5a78b_4362_4dbf_8159_875c6b827c25.slice/crio-78b70ea7a7a24978005d5a53837ff1ee300a063a579efe57efe7b7f11521a0d2 WatchSource:0}: Error finding container 78b70ea7a7a24978005d5a53837ff1ee300a063a579efe57efe7b7f11521a0d2: Status 404 returned error can't find the container with id 78b70ea7a7a24978005d5a53837ff1ee300a063a579efe57efe7b7f11521a0d2
Jul 27 23:55:03.345231 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.345196    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"]
Jul 27 23:55:03.347229 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:55:03.347196    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede34508_011f_41af_8f7e_35332581d005.slice/crio-d0d78b586c4c62069a6ec25c050e28e445288ffc69d03070dc131b6c27648c2b WatchSource:0}: Error finding container d0d78b586c4c62069a6ec25c050e28e445288ffc69d03070dc131b6c27648c2b: Status 404 returned error can't find the container with id d0d78b586c4c62069a6ec25c050e28e445288ffc69d03070dc131b6c27648c2b
Jul 27 23:55:03.375715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:03.375685    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"]
Jul 27 23:55:03.385547 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:55:03.385519    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fde4ea_07a1_4d11_9b14_ab3836d4552c.slice/crio-0af6d0308844802d7f47c4cfaed36a4b9d3eef4de8615470fcf16b5761ee0dd5 WatchSource:0}: Error finding container 0af6d0308844802d7f47c4cfaed36a4b9d3eef4de8615470fcf16b5761ee0dd5: Status 404 returned error can't find the container with id 0af6d0308844802d7f47c4cfaed36a4b9d3eef4de8615470fcf16b5761ee0dd5
Jul 27 23:55:04.163013 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.162954    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7" event={"ID":"ede34508-011f-41af-8f7e-35332581d005","Type":"ContainerStarted","Data":"98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d"}
Jul 27 23:55:04.163219 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.163039    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:04.163219 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.163064    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7" event={"ID":"ede34508-011f-41af-8f7e-35332581d005","Type":"ContainerStarted","Data":"d0d78b586c4c62069a6ec25c050e28e445288ffc69d03070dc131b6c27648c2b"}
Jul 27 23:55:04.164514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.164483    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf" event={"ID":"e8fde4ea-07a1-4d11-9b14-ab3836d4552c","Type":"ContainerStarted","Data":"2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000"}
Jul 27 23:55:04.164658 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.164541    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:04.164658 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.164567    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf" event={"ID":"e8fde4ea-07a1-4d11-9b14-ab3836d4552c","Type":"ContainerStarted","Data":"0af6d0308844802d7f47c4cfaed36a4b9d3eef4de8615470fcf16b5761ee0dd5"}
Jul 27 23:55:04.166252 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.166226    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" event={"ID":"6cd5a78b-4362-4dbf-8159-875c6b827c25","Type":"ContainerStarted","Data":"b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e"}
Jul 27 23:55:04.166385 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.166272    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" event={"ID":"6cd5a78b-4362-4dbf-8159-875c6b827c25","Type":"ContainerStarted","Data":"78b70ea7a7a24978005d5a53837ff1ee300a063a579efe57efe7b7f11521a0d2"}
Jul 27 23:55:04.166385 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.166301    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:04.181540 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.181490    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7" podStartSLOduration=2.181474304 podStartE2EDuration="2.181474304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:04.179508697 +0000 UTC m=+2329.130852106" watchObservedRunningTime="2026-07-27 23:55:04.181474304 +0000 UTC m=+2329.132817731"
Jul 27 23:55:04.196276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.196214    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf" podStartSLOduration=2.196192079 podStartE2EDuration="2.196192079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:04.194143566 +0000 UTC m=+2329.145486969" watchObservedRunningTime="2026-07-27 23:55:04.196192079 +0000 UTC m=+2329.147535468"
Jul 27 23:55:04.212760 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:04.212705    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" podStartSLOduration=1.715191895 podStartE2EDuration="2.212687676s" totalImagesPullingTime="497.495781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:04.211439442 +0000 UTC m=+2329.162782854" watchObservedRunningTime="2026-07-27 23:55:04.212687676 +0000 UTC m=+2329.164031108"
Jul 27 23:55:10.182131 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.182043    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:55:10.553378 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.553267    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"]
Jul 27 23:55:10.567940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.567910    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"]
Jul 27 23:55:10.568283 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.568220    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.573935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.573914    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3ed4f0-serving-cert\""
Jul 27 23:55:10.656907 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.656853    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/108e5577-812b-4700-a465-dc4f732748fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.657123 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.656950    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68ld\" (UniqueName: \"kubernetes.io/projected/108e5577-812b-4700-a465-dc4f732748fb-kube-api-access-c68ld\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.657123 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.656989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/108e5577-812b-4700-a465-dc4f732748fb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.758135 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.757597    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/108e5577-812b-4700-a465-dc4f732748fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.758316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.758233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68ld\" (UniqueName: \"kubernetes.io/projected/108e5577-812b-4700-a465-dc4f732748fb-kube-api-access-c68ld\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.758316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.758291    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/108e5577-812b-4700-a465-dc4f732748fb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.759338 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.759303    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/108e5577-812b-4700-a465-dc4f732748fb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.767386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.760909    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/108e5577-812b-4700-a465-dc4f732748fb-maas-api-tls\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.767670 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.767620    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68ld\" (UniqueName: \"kubernetes.io/projected/108e5577-812b-4700-a465-dc4f732748fb-kube-api-access-c68ld\") pod \"maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:10.896036 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:10.896002    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:11.082687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.082651    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"]
Jul 27 23:55:11.087406 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:55:11.087366    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108e5577_812b_4700_a465_dc4f732748fb.slice/crio-7e1d97631b897b70f6bdac987bacfc7c8378e750409b3a48ad32c4a075e6b6a2 WatchSource:0}: Error finding container 7e1d97631b897b70f6bdac987bacfc7c8378e750409b3a48ad32c4a075e6b6a2: Status 404 returned error can't find the container with id 7e1d97631b897b70f6bdac987bacfc7c8378e750409b3a48ad32c4a075e6b6a2
Jul 27 23:55:11.204552 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.204453    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl" event={"ID":"108e5577-812b-4700-a465-dc4f732748fb","Type":"ContainerStarted","Data":"7e1d97631b897b70f6bdac987bacfc7c8378e750409b3a48ad32c4a075e6b6a2"}
Jul 27 23:55:11.365105 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.365069    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"]
Jul 27 23:55:11.369806 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.369789    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.372818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.372785    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3ed4f0\""
Jul 27 23:55:11.372961 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.372891    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3ed4f0-dockercfg-jfnpl\""
Jul 27 23:55:11.379736 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.378503    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"]
Jul 27 23:55:11.395818 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.395779    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"]
Jul 27 23:55:11.399329 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.399308    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.410568 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.410333    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"]
Jul 27 23:55:11.478844 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.478813    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d96129ff-508a-482e-8387-41cfa82b0306-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.478969 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.478909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnng\" (UniqueName: \"kubernetes.io/projected/d96129ff-508a-482e-8387-41cfa82b0306-kube-api-access-dpnng\") pod \"payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.478969 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.478954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38602fb-4278-49cf-9850-e8142f6273b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.479120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.479103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cl5c\" (UniqueName: \"kubernetes.io/projected/c38602fb-4278-49cf-9850-e8142f6273b7-kube-api-access-6cl5c\") pod \"payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.580314 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.580275    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cl5c\" (UniqueName: \"kubernetes.io/projected/c38602fb-4278-49cf-9850-e8142f6273b7-kube-api-access-6cl5c\") pod \"payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.580512 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.580398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d96129ff-508a-482e-8387-41cfa82b0306-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.580512 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.580453    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnng\" (UniqueName: \"kubernetes.io/projected/d96129ff-508a-482e-8387-41cfa82b0306-kube-api-access-dpnng\") pod \"payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.580512 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.580489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38602fb-4278-49cf-9850-e8142f6273b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.581232 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.581209    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38602fb-4278-49cf-9850-e8142f6273b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.581327 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.581291    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d96129ff-508a-482e-8387-41cfa82b0306-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.588273 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.588251    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnng\" (UniqueName: \"kubernetes.io/projected/d96129ff-508a-482e-8387-41cfa82b0306-kube-api-access-dpnng\") pod \"payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.589312 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.589289    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cl5c\" (UniqueName: \"kubernetes.io/projected/c38602fb-4278-49cf-9850-e8142f6273b7-kube-api-access-6cl5c\") pod \"payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.683687 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.683658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:11.712129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.712100    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:11.861151 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.861029    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"]
Jul 27 23:55:11.863843 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:55:11.863804    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38602fb_4278_49cf_9850_e8142f6273b7.slice/crio-5add4004cbde55cf0b02a5de896d2cc5a3d7296917d2e1dea17d331a6a9de359 WatchSource:0}: Error finding container 5add4004cbde55cf0b02a5de896d2cc5a3d7296917d2e1dea17d331a6a9de359: Status 404 returned error can't find the container with id 5add4004cbde55cf0b02a5de896d2cc5a3d7296917d2e1dea17d331a6a9de359
Jul 27 23:55:11.891204 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:11.891175    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"]
Jul 27 23:55:11.895690 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:55:11.895653    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96129ff_508a_482e_8387_41cfa82b0306.slice/crio-253f2cf49b6768d771ae35d8e3ef825bdba2946a54d2703feb31e493212a677e WatchSource:0}: Error finding container 253f2cf49b6768d771ae35d8e3ef825bdba2946a54d2703feb31e493212a677e: Status 404 returned error can't find the container with id 253f2cf49b6768d771ae35d8e3ef825bdba2946a54d2703feb31e493212a677e
Jul 27 23:55:12.212233 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.212189    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl" event={"ID":"108e5577-812b-4700-a465-dc4f732748fb","Type":"ContainerStarted","Data":"2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07"}
Jul 27 23:55:12.212748 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.212396    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:12.214166 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.214037    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5" event={"ID":"d96129ff-508a-482e-8387-41cfa82b0306","Type":"ContainerStarted","Data":"10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3"}
Jul 27 23:55:12.214166 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.214103    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5" event={"ID":"d96129ff-508a-482e-8387-41cfa82b0306","Type":"ContainerStarted","Data":"253f2cf49b6768d771ae35d8e3ef825bdba2946a54d2703feb31e493212a677e"}
Jul 27 23:55:12.214166 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.214138    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:12.215787 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.215754    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8" event={"ID":"c38602fb-4278-49cf-9850-e8142f6273b7","Type":"ContainerStarted","Data":"3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5"}
Jul 27 23:55:12.215934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.215805    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8" event={"ID":"c38602fb-4278-49cf-9850-e8142f6273b7","Type":"ContainerStarted","Data":"5add4004cbde55cf0b02a5de896d2cc5a3d7296917d2e1dea17d331a6a9de359"}
Jul 27 23:55:12.215934 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.215907    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:55:12.230272 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.230153    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl" podStartSLOduration=1.855329534 podStartE2EDuration="2.230132093s" totalImagesPullingTime="374.802559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:12.22949908 +0000 UTC m=+2337.180842487" watchObservedRunningTime="2026-07-27 23:55:12.230132093 +0000 UTC m=+2337.181475482"
Jul 27 23:55:12.248112 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.248046    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8" podStartSLOduration=1.24803035 podStartE2EDuration="1.24803035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:12.246867305 +0000 UTC m=+2337.198210720" watchObservedRunningTime="2026-07-27 23:55:12.24803035 +0000 UTC m=+2337.199373737"
Jul 27 23:55:12.265509 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:12.265446    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5" podStartSLOduration=1.265423574 podStartE2EDuration="1.265423574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:55:12.262935647 +0000 UTC m=+2337.214279034" watchObservedRunningTime="2026-07-27 23:55:12.265423574 +0000 UTC m=+2337.216766981"
Jul 27 23:55:15.174768 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.174736    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:55:15.175249 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.174927    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:55:15.237856 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.237823    2570 generic.go:356] "Generic (PLEG): container finished" podID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerID="c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68" exitCode=0
Jul 27 23:55:15.238050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.237911    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" event={"ID":"58d2211b-a134-4921-b89c-8dfecdb8c865","Type":"ContainerDied","Data":"c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"}
Jul 27 23:55:15.238518 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.238497    2570 scope.go:117] "RemoveContainer" containerID="c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"
Jul 27 23:55:15.239879 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.239842    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:55:15.239956 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.239938    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:55:15.240040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.240018    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:55:15.744833 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:15.744801    2570 scope.go:117] "RemoveContainer" containerID="cd102a3bca103e5ea1376481d4733344f653d8b2e531b25f77dd620c59385167"
Jul 27 23:55:16.146213 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:16.146179    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:16.250997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:16.250952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" event={"ID":"58d2211b-a134-4921-b89c-8dfecdb8c865","Type":"ContainerStarted","Data":"18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c"}
Jul 27 23:55:16.616774 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:16.616677    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:16.618064 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:16.618034    2570 patch_prober.go:28] interesting pod/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 27 23:55:16.618184 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:16.618088    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 27 23:55:17.616805 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:17.616770    2570 patch_prober.go:28] interesting pod/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 27 23:55:17.617191 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:17.616826    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 27 23:55:18.228111 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:18.228079    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:55:18.616042 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:18.615952    2570 patch_prober.go:28] interesting pod/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 27 23:55:18.616042 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:18.616017    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 27 23:55:19.616630 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:19.616587    2570 patch_prober.go:28] interesting pod/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 27 23:55:19.617094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:19.616651    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 27 23:55:20.621172 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:20.621142    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:20.621567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:20.621543    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:20.622181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:20.622163    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:55:23.224675 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:23.224642    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:55:23.225156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:55:23.224730    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:56:15.580082 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:56:15.580056    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:56:15.583292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:56:15.583266    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 27 23:57:02.108026 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.107944    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"]
Jul 27 23:57:02.108562 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.108214    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" podUID="6cd5a78b-4362-4dbf-8159-875c6b827c25" containerName="maas-api" containerID="cri-o://b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e" gracePeriod=30
Jul 27 23:57:02.221146 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.221109    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"]
Jul 27 23:57:02.221438 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.221394    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7" podUID="ede34508-011f-41af-8f7e-35332581d005" containerName="payload-processing" containerID="cri-o://98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d" gracePeriod=30
Jul 27 23:57:02.233226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.233195    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"]
Jul 27 23:57:02.233491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.233456    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf" podUID="e8fde4ea-07a1-4d11-9b14-ab3836d4552c" containerName="payload-pre-processing" containerID="cri-o://2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000" gracePeriod=30
Jul 27 23:57:02.377437 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.377407    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:57:02.407882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.407853    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls\") pod \"6cd5a78b-4362-4dbf-8159-875c6b827c25\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") "
Jul 27 23:57:02.408065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.407911    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd5a78b-4362-4dbf-8159-875c6b827c25-openshift-service-ca\") pod \"6cd5a78b-4362-4dbf-8159-875c6b827c25\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") "
Jul 27 23:57:02.408065 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.407948    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr87p\" (UniqueName: \"kubernetes.io/projected/6cd5a78b-4362-4dbf-8159-875c6b827c25-kube-api-access-hr87p\") pod \"6cd5a78b-4362-4dbf-8159-875c6b827c25\" (UID: \"6cd5a78b-4362-4dbf-8159-875c6b827c25\") "
Jul 27 23:57:02.408390 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.408332    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd5a78b-4362-4dbf-8159-875c6b827c25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6cd5a78b-4362-4dbf-8159-875c6b827c25" (UID: "6cd5a78b-4362-4dbf-8159-875c6b827c25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:02.413018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.412971    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6cd5a78b-4362-4dbf-8159-875c6b827c25" (UID: "6cd5a78b-4362-4dbf-8159-875c6b827c25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:57:02.414580 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.414540    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd5a78b-4362-4dbf-8159-875c6b827c25-kube-api-access-hr87p" (OuterVolumeSpecName: "kube-api-access-hr87p") pod "6cd5a78b-4362-4dbf-8159-875c6b827c25" (UID: "6cd5a78b-4362-4dbf-8159-875c6b827c25"). InnerVolumeSpecName "kube-api-access-hr87p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:02.508758 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.508730    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cd5a78b-4362-4dbf-8159-875c6b827c25-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.508921 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.508762    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd5a78b-4362-4dbf-8159-875c6b827c25-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.508921 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.508780    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr87p\" (UniqueName: \"kubernetes.io/projected/6cd5a78b-4362-4dbf-8159-875c6b827c25-kube-api-access-hr87p\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.516512 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.516488    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:57:02.520514 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.520485    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:57:02.609658 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.609623    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-plugins-config-volume\") pod \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") "
Jul 27 23:57:02.609658 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.609661    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nt2r\" (UniqueName: \"kubernetes.io/projected/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-kube-api-access-7nt2r\") pod \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\" (UID: \"e8fde4ea-07a1-4d11-9b14-ab3836d4552c\") "
Jul 27 23:57:02.609902 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.609685    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede34508-011f-41af-8f7e-35332581d005-plugins-config-volume\") pod \"ede34508-011f-41af-8f7e-35332581d005\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") "
Jul 27 23:57:02.609902 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.609783    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcmdc\" (UniqueName: \"kubernetes.io/projected/ede34508-011f-41af-8f7e-35332581d005-kube-api-access-mcmdc\") pod \"ede34508-011f-41af-8f7e-35332581d005\" (UID: \"ede34508-011f-41af-8f7e-35332581d005\") "
Jul 27 23:57:02.610090 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.610064    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8fde4ea-07a1-4d11-9b14-ab3836d4552c" (UID: "e8fde4ea-07a1-4d11-9b14-ab3836d4552c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:02.610289 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.610100    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede34508-011f-41af-8f7e-35332581d005-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ede34508-011f-41af-8f7e-35332581d005" (UID: "ede34508-011f-41af-8f7e-35332581d005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:02.612047 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.612027    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-kube-api-access-7nt2r" (OuterVolumeSpecName: "kube-api-access-7nt2r") pod "e8fde4ea-07a1-4d11-9b14-ab3836d4552c" (UID: "e8fde4ea-07a1-4d11-9b14-ab3836d4552c"). InnerVolumeSpecName "kube-api-access-7nt2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:02.612132 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.612057    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede34508-011f-41af-8f7e-35332581d005-kube-api-access-mcmdc" (OuterVolumeSpecName: "kube-api-access-mcmdc") pod "ede34508-011f-41af-8f7e-35332581d005" (UID: "ede34508-011f-41af-8f7e-35332581d005"). InnerVolumeSpecName "kube-api-access-mcmdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:02.711462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.711417    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.711462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.711457    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nt2r\" (UniqueName: \"kubernetes.io/projected/e8fde4ea-07a1-4d11-9b14-ab3836d4552c-kube-api-access-7nt2r\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.711462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.711468    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ede34508-011f-41af-8f7e-35332581d005-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.711462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.711477    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmdc\" (UniqueName: \"kubernetes.io/projected/ede34508-011f-41af-8f7e-35332581d005-kube-api-access-mcmdc\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:02.792727 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.792692    2570 generic.go:356] "Generic (PLEG): container finished" podID="ede34508-011f-41af-8f7e-35332581d005" containerID="98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d" exitCode=0
Jul 27 23:57:02.792880 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.792735    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"
Jul 27 23:57:02.792880 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.792767    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7" event={"ID":"ede34508-011f-41af-8f7e-35332581d005","Type":"ContainerDied","Data":"98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d"}
Jul 27 23:57:02.792880 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.792820    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7" event={"ID":"ede34508-011f-41af-8f7e-35332581d005","Type":"ContainerDied","Data":"d0d78b586c4c62069a6ec25c050e28e445288ffc69d03070dc131b6c27648c2b"}
Jul 27 23:57:02.792880 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.792843    2570 scope.go:117] "RemoveContainer" containerID="98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d"
Jul 27 23:57:02.794419 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.794399    2570 generic.go:356] "Generic (PLEG): container finished" podID="e8fde4ea-07a1-4d11-9b14-ab3836d4552c" containerID="2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000" exitCode=0
Jul 27 23:57:02.794548 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.794452    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"
Jul 27 23:57:02.794548 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.794498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf" event={"ID":"e8fde4ea-07a1-4d11-9b14-ab3836d4552c","Type":"ContainerDied","Data":"2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000"}
Jul 27 23:57:02.794664 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.794556    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf" event={"ID":"e8fde4ea-07a1-4d11-9b14-ab3836d4552c","Type":"ContainerDied","Data":"0af6d0308844802d7f47c4cfaed36a4b9d3eef4de8615470fcf16b5761ee0dd5"}
Jul 27 23:57:02.795971 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.795949    2570 generic.go:356] "Generic (PLEG): container finished" podID="6cd5a78b-4362-4dbf-8159-875c6b827c25" containerID="b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e" exitCode=0
Jul 27 23:57:02.796083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.795978    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" event={"ID":"6cd5a78b-4362-4dbf-8159-875c6b827c25","Type":"ContainerDied","Data":"b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e"}
Jul 27 23:57:02.796083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.796005    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"
Jul 27 23:57:02.796083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.796035    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm" event={"ID":"6cd5a78b-4362-4dbf-8159-875c6b827c25","Type":"ContainerDied","Data":"78b70ea7a7a24978005d5a53837ff1ee300a063a579efe57efe7b7f11521a0d2"}
Jul 27 23:57:02.806059 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.805866    2570 scope.go:117] "RemoveContainer" containerID="98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d"
Jul 27 23:57:02.806212 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:02.806180    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d\": container with ID starting with 98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d not found: ID does not exist" containerID="98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d"
Jul 27 23:57:02.806275 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.806214    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d"} err="failed to get container status \"98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d\": rpc error: code = NotFound desc = could not find container \"98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d\": container with ID starting with 98369c264652f34c2a8a8f36f8f9d4e37c239cff5719e8beccb3723f5248e13d not found: ID does not exist"
Jul 27 23:57:02.806275 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.806239    2570 scope.go:117] "RemoveContainer" containerID="2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000"
Jul 27 23:57:02.814479 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.814462    2570 scope.go:117] "RemoveContainer" containerID="2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000"
Jul 27 23:57:02.814742 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:02.814725    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000\": container with ID starting with 2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000 not found: ID does not exist" containerID="2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000"
Jul 27 23:57:02.814807 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.814748    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000"} err="failed to get container status \"2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000\": rpc error: code = NotFound desc = could not find container \"2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000\": container with ID starting with 2b0e76dc9db536bafa7cd0982a9bfa2cc4562d60d6a8ff387205b413dedfa000 not found: ID does not exist"
Jul 27 23:57:02.814807 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.814774    2570 scope.go:117] "RemoveContainer" containerID="b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e"
Jul 27 23:57:02.820954 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.820934    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"]
Jul 27 23:57:02.823234 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.823215    2570 scope.go:117] "RemoveContainer" containerID="b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e"
Jul 27 23:57:02.823539 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:02.823517    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e\": container with ID starting with b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e not found: ID does not exist" containerID="b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e"
Jul 27 23:57:02.823607 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.823546    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e"} err="failed to get container status \"b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e\": rpc error: code = NotFound desc = could not find container \"b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e\": container with ID starting with b2f55bc5412952d56a6aeaf77edae10445071f0562a6f568ca875777027dda6e not found: ID does not exist"
Jul 27 23:57:02.824528 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.824510    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-05a1fc-78b76ddd66-9r2v7"]
Jul 27 23:57:02.834501 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.834482    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"]
Jul 27 23:57:02.838467 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.838449    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-05a1fc-f6469cfc7-4vlcm"]
Jul 27 23:57:02.848292 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.848271    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"]
Jul 27 23:57:02.851633 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:02.851614    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-05a1fc-6bd5fc9764-wxhmf"]
Jul 27 23:57:03.522737 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:03.522708    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd5a78b-4362-4dbf-8159-875c6b827c25" path="/var/lib/kubelet/pods/6cd5a78b-4362-4dbf-8159-875c6b827c25/volumes"
Jul 27 23:57:03.523111 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:03.523061    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fde4ea-07a1-4d11-9b14-ab3836d4552c" path="/var/lib/kubelet/pods/e8fde4ea-07a1-4d11-9b14-ab3836d4552c/volumes"
Jul 27 23:57:03.523388 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:03.523373    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede34508-011f-41af-8f7e-35332581d005" path="/var/lib/kubelet/pods/ede34508-011f-41af-8f7e-35332581d005/volumes"
Jul 27 23:57:13.167652 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:13.167597    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"]
Jul 27 23:57:13.168074 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:13.167907    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy" containerID="cri-o://18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c" gracePeriod=30
Jul 27 23:57:18.428040 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.427972    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:57:18.558504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558468    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-credential-socket\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558512    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mt8\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-kube-api-access-68mt8\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558591    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2211b-a134-4921-b89c-8dfecdb8c865-istiod-ca-cert\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558618    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-socket\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558647    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-token\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558695    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-certs\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558762 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558725    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-envoy\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558772    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-data\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558797    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-podinfo\") pod \"58d2211b-a134-4921-b89c-8dfecdb8c865\" (UID: \"58d2211b-a134-4921-b89c-8dfecdb8c865\") "
Jul 27 23:57:18.558999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558800    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:18.558999 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.558919    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:18.559214 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.559007    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d2211b-a134-4921-b89c-8dfecdb8c865-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:18.559214 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.559110    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:18.559325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.559255    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2211b-a134-4921-b89c-8dfecdb8c865-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.559325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.559277    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.559325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.559293    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.559325 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.559307    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.560085 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.560054    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-data" (OuterVolumeSpecName: "istio-data") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:18.561117 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.561089    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:57:18.561117 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.561105    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-token" (OuterVolumeSpecName: "istio-token") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:18.561285 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.561115    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-kube-api-access-68mt8" (OuterVolumeSpecName: "kube-api-access-68mt8") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "kube-api-access-68mt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:18.561285 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.561178    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58d2211b-a134-4921-b89c-8dfecdb8c865" (UID: "58d2211b-a134-4921-b89c-8dfecdb8c865"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:57:18.660546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.660509    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mt8\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-kube-api-access-68mt8\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.660546 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.660545    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.660747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.660560    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.660747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.660571    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.660747 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.660582    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2211b-a134-4921-b89c-8dfecdb8c865-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:18.879077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.878975    2570 generic.go:356] "Generic (PLEG): container finished" podID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerID="18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c" exitCode=0
Jul 27 23:57:18.879077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.879053    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"
Jul 27 23:57:18.879077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.879052    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" event={"ID":"58d2211b-a134-4921-b89c-8dfecdb8c865","Type":"ContainerDied","Data":"18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c"}
Jul 27 23:57:18.879395 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.879110    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz" event={"ID":"58d2211b-a134-4921-b89c-8dfecdb8c865","Type":"ContainerDied","Data":"578752d4c47d09a681a17168850bc669d82f1719079f94f8d0e11f4ed2c8b8ad"}
Jul 27 23:57:18.879395 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.879131    2570 scope.go:117] "RemoveContainer" containerID="18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c"
Jul 27 23:57:18.888310 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.888292    2570 scope.go:117] "RemoveContainer" containerID="c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"
Jul 27 23:57:18.896441 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.896425    2570 scope.go:117] "RemoveContainer" containerID="18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c"
Jul 27 23:57:18.896671 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:18.896655    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c\": container with ID starting with 18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c not found: ID does not exist" containerID="18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c"
Jul 27 23:57:18.896715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.896680    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c"} err="failed to get container status \"18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c\": rpc error: code = NotFound desc = could not find container \"18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c\": container with ID starting with 18abcbdd9049f98d290b5173aab96ea26867a457d0654062cd826e70f799aa6c not found: ID does not exist"
Jul 27 23:57:18.896715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.896698    2570 scope.go:117] "RemoveContainer" containerID="c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"
Jul 27 23:57:18.896954 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:18.896933    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68\": container with ID starting with c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68 not found: ID does not exist" containerID="c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"
Jul 27 23:57:18.896993 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.896963    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68"} err="failed to get container status \"c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68\": rpc error: code = NotFound desc = could not find container \"c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68\": container with ID starting with c8ee9bb80b0a99c95bf329e23b695c6a29df08be1847233b54186619001f2f68 not found: ID does not exist"
Jul 27 23:57:18.900625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.900603    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"]
Jul 27 23:57:18.904451 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:18.904429    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-05a1fc-openshift-default-7566b87fd-2g7mz"]
Jul 27 23:57:19.522276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:19.522238    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" path="/var/lib/kubelet/pods/58d2211b-a134-4921-b89c-8dfecdb8c865/volumes"
Jul 27 23:57:24.052918 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.052882    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"]
Jul 27 23:57:24.053328 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.053107    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl" podUID="108e5577-812b-4700-a465-dc4f732748fb" containerName="maas-api" containerID="cri-o://2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07" gracePeriod=30
Jul 27 23:57:24.098002 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.097972    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"]
Jul 27 23:57:24.098201 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.098178    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5" podUID="d96129ff-508a-482e-8387-41cfa82b0306" containerName="payload-processing" containerID="cri-o://10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3" gracePeriod=30
Jul 27 23:57:24.106719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.106692    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"]
Jul 27 23:57:24.106974 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.106939    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8" podUID="c38602fb-4278-49cf-9850-e8142f6273b7" containerName="payload-pre-processing" containerID="cri-o://3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5" gracePeriod=30
Jul 27 23:57:24.368192 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.368163    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:57:24.377425 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.377400    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:57:24.411854 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.411829    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:57:24.515734 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515692    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpnng\" (UniqueName: \"kubernetes.io/projected/d96129ff-508a-482e-8387-41cfa82b0306-kube-api-access-dpnng\") pod \"d96129ff-508a-482e-8387-41cfa82b0306\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") "
Jul 27 23:57:24.515941 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515751    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/108e5577-812b-4700-a465-dc4f732748fb-openshift-service-ca\") pod \"108e5577-812b-4700-a465-dc4f732748fb\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") "
Jul 27 23:57:24.515941 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515774    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cl5c\" (UniqueName: \"kubernetes.io/projected/c38602fb-4278-49cf-9850-e8142f6273b7-kube-api-access-6cl5c\") pod \"c38602fb-4278-49cf-9850-e8142f6273b7\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") "
Jul 27 23:57:24.515941 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515804    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38602fb-4278-49cf-9850-e8142f6273b7-plugins-config-volume\") pod \"c38602fb-4278-49cf-9850-e8142f6273b7\" (UID: \"c38602fb-4278-49cf-9850-e8142f6273b7\") "
Jul 27 23:57:24.515941 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515846    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d96129ff-508a-482e-8387-41cfa82b0306-plugins-config-volume\") pod \"d96129ff-508a-482e-8387-41cfa82b0306\" (UID: \"d96129ff-508a-482e-8387-41cfa82b0306\") "
Jul 27 23:57:24.516150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515947    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c68ld\" (UniqueName: \"kubernetes.io/projected/108e5577-812b-4700-a465-dc4f732748fb-kube-api-access-c68ld\") pod \"108e5577-812b-4700-a465-dc4f732748fb\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") "
Jul 27 23:57:24.516150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.515983    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/108e5577-812b-4700-a465-dc4f732748fb-maas-api-tls\") pod \"108e5577-812b-4700-a465-dc4f732748fb\" (UID: \"108e5577-812b-4700-a465-dc4f732748fb\") "
Jul 27 23:57:24.516237 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.516159    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108e5577-812b-4700-a465-dc4f732748fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "108e5577-812b-4700-a465-dc4f732748fb" (UID: "108e5577-812b-4700-a465-dc4f732748fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:24.516237 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.516178    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38602fb-4278-49cf-9850-e8142f6273b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c38602fb-4278-49cf-9850-e8142f6273b7" (UID: "c38602fb-4278-49cf-9850-e8142f6273b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:24.516327 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.516264    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96129ff-508a-482e-8387-41cfa82b0306-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d96129ff-508a-482e-8387-41cfa82b0306" (UID: "d96129ff-508a-482e-8387-41cfa82b0306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:57:24.516327 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.516286    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/108e5577-812b-4700-a465-dc4f732748fb-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.516327 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.516302    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c38602fb-4278-49cf-9850-e8142f6273b7-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.518152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.518119    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108e5577-812b-4700-a465-dc4f732748fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "108e5577-812b-4700-a465-dc4f732748fb" (UID: "108e5577-812b-4700-a465-dc4f732748fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:57:24.518152 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.518153    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38602fb-4278-49cf-9850-e8142f6273b7-kube-api-access-6cl5c" (OuterVolumeSpecName: "kube-api-access-6cl5c") pod "c38602fb-4278-49cf-9850-e8142f6273b7" (UID: "c38602fb-4278-49cf-9850-e8142f6273b7"). InnerVolumeSpecName "kube-api-access-6cl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:24.518348 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.518133    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96129ff-508a-482e-8387-41cfa82b0306-kube-api-access-dpnng" (OuterVolumeSpecName: "kube-api-access-dpnng") pod "d96129ff-508a-482e-8387-41cfa82b0306" (UID: "d96129ff-508a-482e-8387-41cfa82b0306"). InnerVolumeSpecName "kube-api-access-dpnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:24.518503 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.518484    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108e5577-812b-4700-a465-dc4f732748fb-kube-api-access-c68ld" (OuterVolumeSpecName: "kube-api-access-c68ld") pod "108e5577-812b-4700-a465-dc4f732748fb" (UID: "108e5577-812b-4700-a465-dc4f732748fb"). InnerVolumeSpecName "kube-api-access-c68ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:57:24.617400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.617271    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c68ld\" (UniqueName: \"kubernetes.io/projected/108e5577-812b-4700-a465-dc4f732748fb-kube-api-access-c68ld\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.617400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.617306    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/108e5577-812b-4700-a465-dc4f732748fb-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.617400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.617316    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnng\" (UniqueName: \"kubernetes.io/projected/d96129ff-508a-482e-8387-41cfa82b0306-kube-api-access-dpnng\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.617400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.617326    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cl5c\" (UniqueName: \"kubernetes.io/projected/c38602fb-4278-49cf-9850-e8142f6273b7-kube-api-access-6cl5c\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.617400 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.617335    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d96129ff-508a-482e-8387-41cfa82b0306-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:57:24.910096 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.910058    2570 generic.go:356] "Generic (PLEG): container finished" podID="d96129ff-508a-482e-8387-41cfa82b0306" containerID="10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3" exitCode=0
Jul 27 23:57:24.910298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.910135    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"
Jul 27 23:57:24.910298 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.910135    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5" event={"ID":"d96129ff-508a-482e-8387-41cfa82b0306","Type":"ContainerDied","Data":"10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3"}
Jul 27 23:57:24.910442 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.910316    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5" event={"ID":"d96129ff-508a-482e-8387-41cfa82b0306","Type":"ContainerDied","Data":"253f2cf49b6768d771ae35d8e3ef825bdba2946a54d2703feb31e493212a677e"}
Jul 27 23:57:24.910442 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.910364    2570 scope.go:117] "RemoveContainer" containerID="10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3"
Jul 27 23:57:24.912083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.912058    2570 generic.go:356] "Generic (PLEG): container finished" podID="c38602fb-4278-49cf-9850-e8142f6273b7" containerID="3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5" exitCode=0
Jul 27 23:57:24.912181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.912123    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"
Jul 27 23:57:24.912181 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.912141    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8" event={"ID":"c38602fb-4278-49cf-9850-e8142f6273b7","Type":"ContainerDied","Data":"3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5"}
Jul 27 23:57:24.912316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.912187    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8" event={"ID":"c38602fb-4278-49cf-9850-e8142f6273b7","Type":"ContainerDied","Data":"5add4004cbde55cf0b02a5de896d2cc5a3d7296917d2e1dea17d331a6a9de359"}
Jul 27 23:57:24.913677 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.913655    2570 generic.go:356] "Generic (PLEG): container finished" podID="108e5577-812b-4700-a465-dc4f732748fb" containerID="2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07" exitCode=0
Jul 27 23:57:24.913803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.913704    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl" event={"ID":"108e5577-812b-4700-a465-dc4f732748fb","Type":"ContainerDied","Data":"2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07"}
Jul 27 23:57:24.913803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.913731    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl" event={"ID":"108e5577-812b-4700-a465-dc4f732748fb","Type":"ContainerDied","Data":"7e1d97631b897b70f6bdac987bacfc7c8378e750409b3a48ad32c4a075e6b6a2"}
Jul 27 23:57:24.913803 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.913747    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"
Jul 27 23:57:24.920296 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.920276    2570 scope.go:117] "RemoveContainer" containerID="10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3"
Jul 27 23:57:24.920608 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:24.920584    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3\": container with ID starting with 10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3 not found: ID does not exist" containerID="10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3"
Jul 27 23:57:24.920690 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.920619    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3"} err="failed to get container status \"10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3\": rpc error: code = NotFound desc = could not find container \"10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3\": container with ID starting with 10c5719c4b3071fb24c3d34bdb87cad2680b4cae00f293168b7e3debd810d1a3 not found: ID does not exist"
Jul 27 23:57:24.920690 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.920642    2570 scope.go:117] "RemoveContainer" containerID="3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5"
Jul 27 23:57:24.929555 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.929534    2570 scope.go:117] "RemoveContainer" containerID="3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5"
Jul 27 23:57:24.929835 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:24.929816    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5\": container with ID starting with 3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5 not found: ID does not exist" containerID="3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5"
Jul 27 23:57:24.929891 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.929843    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5"} err="failed to get container status \"3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5\": rpc error: code = NotFound desc = could not find container \"3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5\": container with ID starting with 3e7b5f3836c8e8b666cd8c7f13320c034cb463de09cf2a34eecad33fa6570ac5 not found: ID does not exist"
Jul 27 23:57:24.929891 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.929860    2570 scope.go:117] "RemoveContainer" containerID="2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07"
Jul 27 23:57:24.937980 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.937925    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"]
Jul 27 23:57:24.939416 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.939348    2570 scope.go:117] "RemoveContainer" containerID="2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07"
Jul 27 23:57:24.939701 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:24.939682    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07\": container with ID starting with 2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07 not found: ID does not exist" containerID="2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07"
Jul 27 23:57:24.939755 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.939709    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07"} err="failed to get container status \"2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07\": rpc error: code = NotFound desc = could not find container \"2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07\": container with ID starting with 2c91dc4d7038ddcc1cf021d9a36016bdd8ae18bd735b6cefda22edc212fc4e07 not found: ID does not exist"
Jul 27 23:57:24.940224 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.940206    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3ed4f0-6c97bbd8d8-76vh5"]
Jul 27 23:57:24.950487 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.950457    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"]
Jul 27 23:57:24.952116 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.952097    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3ed4f0-7d7894784-d7mg8"]
Jul 27 23:57:24.962156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.962115    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"]
Jul 27 23:57:24.965966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:24.965939    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3ed4f0-5c64b94dc8-t4fjl"]
Jul 27 23:57:25.522947 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:25.522911    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="108e5577-812b-4700-a465-dc4f732748fb" path="/var/lib/kubelet/pods/108e5577-812b-4700-a465-dc4f732748fb/volumes"
Jul 27 23:57:25.523330 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:25.523257    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38602fb-4278-49cf-9850-e8142f6273b7" path="/var/lib/kubelet/pods/c38602fb-4278-49cf-9850-e8142f6273b7/volumes"
Jul 27 23:57:25.523583 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:25.523566    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d96129ff-508a-482e-8387-41cfa82b0306" path="/var/lib/kubelet/pods/d96129ff-508a-482e-8387-41cfa82b0306/volumes"
Jul 27 23:57:35.754450 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754412    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"]
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754816    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38602fb-4278-49cf-9850-e8142f6273b7" containerName="payload-pre-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754828    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38602fb-4278-49cf-9850-e8142f6273b7" containerName="payload-pre-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754845    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108e5577-812b-4700-a465-dc4f732748fb" containerName="maas-api"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754850    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="108e5577-812b-4700-a465-dc4f732748fb" containerName="maas-api"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754862    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754868    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754877    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754884    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754889    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d96129ff-508a-482e-8387-41cfa82b0306" containerName="payload-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754895    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d96129ff-508a-482e-8387-41cfa82b0306" containerName="payload-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754902    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fde4ea-07a1-4d11-9b14-ab3836d4552c" containerName="payload-pre-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754907    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fde4ea-07a1-4d11-9b14-ab3836d4552c" containerName="payload-pre-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754917    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede34508-011f-41af-8f7e-35332581d005" containerName="payload-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754922    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede34508-011f-41af-8f7e-35332581d005" containerName="payload-processing"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754929    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd5a78b-4362-4dbf-8159-875c6b827c25" containerName="maas-api"
Jul 27 23:57:35.754927 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.754933    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd5a78b-4362-4dbf-8159-875c6b827c25" containerName="maas-api"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755011    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755019    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fde4ea-07a1-4d11-9b14-ab3836d4552c" containerName="payload-pre-processing"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755028    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede34508-011f-41af-8f7e-35332581d005" containerName="payload-processing"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755038    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cd5a78b-4362-4dbf-8159-875c6b827c25" containerName="maas-api"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755046    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38602fb-4278-49cf-9850-e8142f6273b7" containerName="payload-pre-processing"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755055    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d96129ff-508a-482e-8387-41cfa82b0306" containerName="payload-processing"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755064    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d2211b-a134-4921-b89c-8dfecdb8c865" containerName="istio-proxy"
Jul 27 23:57:35.755761 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.755072    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="108e5577-812b-4700-a465-dc4f732748fb" containerName="maas-api"
Jul 27 23:57:35.759985 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.759964    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.762539 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.762513    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f3414a-openshift-default-dockercfg-f8hg7\""
Jul 27 23:57:35.768850 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.768824    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"]
Jul 27 23:57:35.921663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921623    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvlc7\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-kube-api-access-lvlc7\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.921663 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921670    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c228a866-c04b-4c21-802c-5b409e6e1b64-istiod-ca-cert\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.921888 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921749    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-podinfo\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.921888 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921816    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-data\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.921888 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921837    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-credential-socket\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.921888 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921872    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-certs\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.922030 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921903    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-socket\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.922030 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-envoy\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:35.922030 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:35.921943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-token\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023098 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-socket\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023098 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023055    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-envoy\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023098 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023081    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-token\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023110    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvlc7\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-kube-api-access-lvlc7\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c228a866-c04b-4c21-802c-5b409e6e1b64-istiod-ca-cert\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023174    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-podinfo\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-data\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-credential-socket\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023391 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023303    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-certs\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023709 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023471    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-socket\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023709 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023660    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-data\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023709 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023690    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-certs\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023845 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023797    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-credential-socket\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.023909 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.023890    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c228a866-c04b-4c21-802c-5b409e6e1b64-istiod-ca-cert\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.026205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.026177    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-envoy\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.026346 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.026326    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-podinfo\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.031259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.031236    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-token\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.031259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.031256    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvlc7\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-kube-api-access-lvlc7\") pod \"e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") " pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.072182 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.072151    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:36.211148 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.211123    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"]
Jul 27 23:57:36.212752 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:36.212711    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc228a866_c04b_4c21_802c_5b409e6e1b64.slice/crio-c1098c08ddaea6ee0e1d0b1a243f86e9cdb503839dc12ecf7d173bbb92cf31a6 WatchSource:0}: Error finding container c1098c08ddaea6ee0e1d0b1a243f86e9cdb503839dc12ecf7d173bbb92cf31a6: Status 404 returned error can't find the container with id c1098c08ddaea6ee0e1d0b1a243f86e9cdb503839dc12ecf7d173bbb92cf31a6
Jul 27 23:57:36.214910 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.214868    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:57:36.215015 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.214955    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:57:36.215059 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.215023    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:57:36.975525 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.975484    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" event={"ID":"c228a866-c04b-4c21-802c-5b409e6e1b64","Type":"ContainerStarted","Data":"429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"}
Jul 27 23:57:36.975905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.975546    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" event={"ID":"c228a866-c04b-4c21-802c-5b409e6e1b64","Type":"ContainerStarted","Data":"c1098c08ddaea6ee0e1d0b1a243f86e9cdb503839dc12ecf7d173bbb92cf31a6"}
Jul 27 23:57:36.995947 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:36.995895    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podStartSLOduration=1.995877833 podStartE2EDuration="1.995877833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:36.994952151 +0000 UTC m=+2481.946295555" watchObservedRunningTime="2026-07-27 23:57:36.995877833 +0000 UTC m=+2481.947221220"
Jul 27 23:57:37.072444 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:37.072406    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:37.077316 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:37.077289    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:37.980532 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:37.980501    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:37.981579 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:37.981561    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:38.926345 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.926311    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"]
Jul 27 23:57:38.930221 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.930197    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:38.932894 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.932869    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f3414a-serving-cert\""
Jul 27 23:57:38.941502 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.941477    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"]
Jul 27 23:57:38.949617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.949584    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6kpm\" (UniqueName: \"kubernetes.io/projected/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-kube-api-access-w6kpm\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:38.949752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.949640    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:38.949752 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:38.949724    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.050526 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.050486    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kpm\" (UniqueName: \"kubernetes.io/projected/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-kube-api-access-w6kpm\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.050526 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.050522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.051050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.050553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.051050 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:39.050707    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-serving-cert: secret "maas-api-e2e-api-a-f3414a-serving-cert" not found
Jul 27 23:57:39.051050 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:39.050786    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls podName:af4dcf1f-a47c-4076-95cd-c4083c8a9dd8 nodeName:}" failed. No retries permitted until 2026-07-27 23:57:39.550763944 +0000 UTC m=+2484.502107311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls") pod "maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" (UID: "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8") : secret "maas-api-e2e-api-a-f3414a-serving-cert" not found
Jul 27 23:57:39.051202 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.051181    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.064321 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.064294    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kpm\" (UniqueName: \"kubernetes.io/projected/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-kube-api-access-w6kpm\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.553362 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.553326    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.556034 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.556005    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls\") pod \"maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.705734 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.705699    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"]
Jul 27 23:57:39.709567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.709547    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.712295 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.712267    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f3414a-dockercfg-ztg5q\""
Jul 27 23:57:39.712454 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.712272    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f3414a\""
Jul 27 23:57:39.719153 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.719126    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"]
Jul 27 23:57:39.753647 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.753608    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"]
Jul 27 23:57:39.754241 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.754209    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a6a5b6-d01b-4dcd-a670-39f486fcb993-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.754412 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.754394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk45\" (UniqueName: \"kubernetes.io/projected/27a6a5b6-d01b-4dcd-a670-39f486fcb993-kube-api-access-kpk45\") pod \"payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.757296 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.757275    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:39.767245 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.767219    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"]
Jul 27 23:57:39.844644 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.844561    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:39.855132 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.855099    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk45\" (UniqueName: \"kubernetes.io/projected/27a6a5b6-d01b-4dcd-a670-39f486fcb993-kube-api-access-kpk45\") pod \"payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.855279 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.855146    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764296fd-fc01-4e14-9d72-300bc3708382-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:39.855279 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.855181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a6a5b6-d01b-4dcd-a670-39f486fcb993-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.855279 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.855210    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjfg8\" (UniqueName: \"kubernetes.io/projected/764296fd-fc01-4e14-9d72-300bc3708382-kube-api-access-fjfg8\") pod \"payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:39.855971 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.855950    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a6a5b6-d01b-4dcd-a670-39f486fcb993-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.864113 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.864080    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk45\" (UniqueName: \"kubernetes.io/projected/27a6a5b6-d01b-4dcd-a670-39f486fcb993-kube-api-access-kpk45\") pod \"payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:39.956436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.955997    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764296fd-fc01-4e14-9d72-300bc3708382-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:39.956436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.956056    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjfg8\" (UniqueName: \"kubernetes.io/projected/764296fd-fc01-4e14-9d72-300bc3708382-kube-api-access-fjfg8\") pod \"payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:39.956935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.956911    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764296fd-fc01-4e14-9d72-300bc3708382-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:39.964510 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:39.964487    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjfg8\" (UniqueName: \"kubernetes.io/projected/764296fd-fc01-4e14-9d72-300bc3708382-kube-api-access-fjfg8\") pod \"payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") " pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:40.021556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.021521    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:40.067935 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.067902    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:40.168480 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.168281    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"]
Jul 27 23:57:40.170271 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:40.170232    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a6a5b6_d01b_4dcd_a670_39f486fcb993.slice/crio-22640e494bc35a01f557a684fa1ecbe9230a94ed308a3d99ab00afca4e21438e WatchSource:0}: Error finding container 22640e494bc35a01f557a684fa1ecbe9230a94ed308a3d99ab00afca4e21438e: Status 404 returned error can't find the container with id 22640e494bc35a01f557a684fa1ecbe9230a94ed308a3d99ab00afca4e21438e
Jul 27 23:57:40.194379 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.194322    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"]
Jul 27 23:57:40.195958 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:40.195913    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4dcf1f_a47c_4076_95cd_c4083c8a9dd8.slice/crio-d491a8ada828de42c982a9b48e976c7a025e0b1c2f111c6786d992282f2aef7a WatchSource:0}: Error finding container d491a8ada828de42c982a9b48e976c7a025e0b1c2f111c6786d992282f2aef7a: Status 404 returned error can't find the container with id d491a8ada828de42c982a9b48e976c7a025e0b1c2f111c6786d992282f2aef7a
Jul 27 23:57:40.197791 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.197772    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 23:57:40.228849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.228824    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"]
Jul 27 23:57:40.235536 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:40.235509    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764296fd_fc01_4e14_9d72_300bc3708382.slice/crio-dec90729b63aeeee0cdd0debb4ead6d08ed05e0cc657ab9caf5c2cf5243c5d8c WatchSource:0}: Error finding container dec90729b63aeeee0cdd0debb4ead6d08ed05e0cc657ab9caf5c2cf5243c5d8c: Status 404 returned error can't find the container with id dec90729b63aeeee0cdd0debb4ead6d08ed05e0cc657ab9caf5c2cf5243c5d8c
Jul 27 23:57:40.997520 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.997483    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp" event={"ID":"27a6a5b6-d01b-4dcd-a670-39f486fcb993","Type":"ContainerStarted","Data":"4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d"}
Jul 27 23:57:40.997700 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.997577    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp" event={"ID":"27a6a5b6-d01b-4dcd-a670-39f486fcb993","Type":"ContainerStarted","Data":"22640e494bc35a01f557a684fa1ecbe9230a94ed308a3d99ab00afca4e21438e"}
Jul 27 23:57:40.997700 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.997627    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:40.999277 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.999242    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" event={"ID":"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8","Type":"ContainerStarted","Data":"95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf"}
Jul 27 23:57:40.999430 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.999297    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" event={"ID":"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8","Type":"ContainerStarted","Data":"d491a8ada828de42c982a9b48e976c7a025e0b1c2f111c6786d992282f2aef7a"}
Jul 27 23:57:40.999430 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:40.999370    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:41.000813 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:41.000790    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7" event={"ID":"764296fd-fc01-4e14-9d72-300bc3708382","Type":"ContainerStarted","Data":"a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e"}
Jul 27 23:57:41.000905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:41.000831    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7" event={"ID":"764296fd-fc01-4e14-9d72-300bc3708382","Type":"ContainerStarted","Data":"dec90729b63aeeee0cdd0debb4ead6d08ed05e0cc657ab9caf5c2cf5243c5d8c"}
Jul 27 23:57:41.000905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:41.000876    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:41.013627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:41.013585    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp" podStartSLOduration=2.013570144 podStartE2EDuration="2.013570144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:41.012922725 +0000 UTC m=+2485.964266113" watchObservedRunningTime="2026-07-27 23:57:41.013570144 +0000 UTC m=+2485.964913532"
Jul 27 23:57:41.029416 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:41.029374    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7" podStartSLOduration=2.02933809 podStartE2EDuration="2.02933809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:41.027457131 +0000 UTC m=+2485.978800548" watchObservedRunningTime="2026-07-27 23:57:41.02933809 +0000 UTC m=+2485.980681454"
Jul 27 23:57:41.044024 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:41.043978    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" podStartSLOduration=2.611570162 podStartE2EDuration="3.043961459s" totalImagesPullingTime="432.391297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:41.042863691 +0000 UTC m=+2485.994207261" watchObservedRunningTime="2026-07-27 23:57:41.043961459 +0000 UTC m=+2485.995304846"
Jul 27 23:57:47.011700 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.011669    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:57:47.496587 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.496556    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"]
Jul 27 23:57:47.504581 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.504552    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.507499 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.507240    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5d597b-serving-cert\""
Jul 27 23:57:47.508399 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.508340    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"]
Jul 27 23:57:47.625771 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.625738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-openshift-service-ca\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.625955 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.625780    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvcb\" (UniqueName: \"kubernetes.io/projected/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-kube-api-access-8mvcb\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.625955 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.625864    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.727331 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.727295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.727582 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.727436    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-openshift-service-ca\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.727582 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.727475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvcb\" (UniqueName: \"kubernetes.io/projected/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-kube-api-access-8mvcb\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.727582 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:47.727480    2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-serving-cert: secret "maas-api-e2e-api-b-5d597b-serving-cert" not found
Jul 27 23:57:47.727582 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:57:47.727558    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls podName:46dcd98c-0bc1-4396-bdf5-75d9f9f28149 nodeName:}" failed. No retries permitted until 2026-07-27 23:57:48.227543228 +0000 UTC m=+2493.178886593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls") pod "maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" (UID: "46dcd98c-0bc1-4396-bdf5-75d9f9f28149") : secret "maas-api-e2e-api-b-5d597b-serving-cert" not found
Jul 27 23:57:47.728034 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.728014    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-openshift-service-ca\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:47.736570 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:47.736540    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvcb\" (UniqueName: \"kubernetes.io/projected/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-kube-api-access-8mvcb\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:48.212082 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.212047    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"]
Jul 27 23:57:48.215648 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.215632    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.218449 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.218429    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5d597b\""
Jul 27 23:57:48.218566 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.218432    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5d597b-dockercfg-m5q5v\""
Jul 27 23:57:48.224617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.224591    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"]
Jul 27 23:57:48.231156 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.231134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:48.234379 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.234338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls\") pod \"maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:48.234837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.234818    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"]
Jul 27 23:57:48.238726 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.238710    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.249720 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.249668    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"]
Jul 27 23:57:48.332547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.332511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486b7d90-9c80-4408-aa7c-d81cded4155a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.332750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.332566    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vv7\" (UniqueName: \"kubernetes.io/projected/a99d9cef-6b64-44d8-91e5-1eee762a0cba-kube-api-access-j9vv7\") pod \"payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") " pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.332750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.332662    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d9cef-6b64-44d8-91e5-1eee762a0cba-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") " pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.332750 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.332704    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gv8\" (UniqueName: \"kubernetes.io/projected/486b7d90-9c80-4408-aa7c-d81cded4155a-kube-api-access-j2gv8\") pod \"payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.418963 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.418928    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:48.433413 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.433384    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486b7d90-9c80-4408-aa7c-d81cded4155a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.433563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.433429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vv7\" (UniqueName: \"kubernetes.io/projected/a99d9cef-6b64-44d8-91e5-1eee762a0cba-kube-api-access-j9vv7\") pod \"payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") " pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.433563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.433471    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d9cef-6b64-44d8-91e5-1eee762a0cba-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") " pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.433563 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.433494    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gv8\" (UniqueName: \"kubernetes.io/projected/486b7d90-9c80-4408-aa7c-d81cded4155a-kube-api-access-j2gv8\") pod \"payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.434247 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.434224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486b7d90-9c80-4408-aa7c-d81cded4155a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.434537 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.434511    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d9cef-6b64-44d8-91e5-1eee762a0cba-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") " pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.442014 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.441988    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gv8\" (UniqueName: \"kubernetes.io/projected/486b7d90-9c80-4408-aa7c-d81cded4155a-kube-api-access-j2gv8\") pod \"payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.442014 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.442002    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vv7\" (UniqueName: \"kubernetes.io/projected/a99d9cef-6b64-44d8-91e5-1eee762a0cba-kube-api-access-j9vv7\") pod \"payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") " pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.527622 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.527550    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:48.582230 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.581921    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:48.616140 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.616105    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"]
Jul 27 23:57:48.620799 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:48.620771    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dcd98c_0bc1_4396_bdf5_75d9f9f28149.slice/crio-0b1c2985fdb323356c49e0bb0f2f65ed3309ef05def03748e7298c49a88fca86 WatchSource:0}: Error finding container 0b1c2985fdb323356c49e0bb0f2f65ed3309ef05def03748e7298c49a88fca86: Status 404 returned error can't find the container with id 0b1c2985fdb323356c49e0bb0f2f65ed3309ef05def03748e7298c49a88fca86
Jul 27 23:57:48.737991 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.737911    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"]
Jul 27 23:57:48.740443 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:48.740400    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486b7d90_9c80_4408_aa7c_d81cded4155a.slice/crio-1df69eb4a5b0dff0f54c266b9a40a6e771981d3e3975cfcb259f9d1d3bbdc090 WatchSource:0}: Error finding container 1df69eb4a5b0dff0f54c266b9a40a6e771981d3e3975cfcb259f9d1d3bbdc090: Status 404 returned error can't find the container with id 1df69eb4a5b0dff0f54c266b9a40a6e771981d3e3975cfcb259f9d1d3bbdc090
Jul 27 23:57:48.797646 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:48.797614    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"]
Jul 27 23:57:48.806589 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:57:48.806557    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99d9cef_6b64_44d8_91e5_1eee762a0cba.slice/crio-bc159e4867474988100c4e9176eab1e0de6a0d22d9f0ae10a7a472e14a2fcf40 WatchSource:0}: Error finding container bc159e4867474988100c4e9176eab1e0de6a0d22d9f0ae10a7a472e14a2fcf40: Status 404 returned error can't find the container with id bc159e4867474988100c4e9176eab1e0de6a0d22d9f0ae10a7a472e14a2fcf40
Jul 27 23:57:49.044003 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.043911    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" event={"ID":"46dcd98c-0bc1-4396-bdf5-75d9f9f28149","Type":"ContainerStarted","Data":"b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f"}
Jul 27 23:57:49.044003 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.043968    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" event={"ID":"46dcd98c-0bc1-4396-bdf5-75d9f9f28149","Type":"ContainerStarted","Data":"0b1c2985fdb323356c49e0bb0f2f65ed3309ef05def03748e7298c49a88fca86"}
Jul 27 23:57:49.044226 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.044022    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:49.045641 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.045618    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6" event={"ID":"a99d9cef-6b64-44d8-91e5-1eee762a0cba","Type":"ContainerStarted","Data":"dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474"}
Jul 27 23:57:49.045766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.045671    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6" event={"ID":"a99d9cef-6b64-44d8-91e5-1eee762a0cba","Type":"ContainerStarted","Data":"bc159e4867474988100c4e9176eab1e0de6a0d22d9f0ae10a7a472e14a2fcf40"}
Jul 27 23:57:49.045766 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.045700    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:57:49.047021 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.046997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n" event={"ID":"486b7d90-9c80-4408-aa7c-d81cded4155a","Type":"ContainerStarted","Data":"71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31"}
Jul 27 23:57:49.047108 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.047036    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n" event={"ID":"486b7d90-9c80-4408-aa7c-d81cded4155a","Type":"ContainerStarted","Data":"1df69eb4a5b0dff0f54c266b9a40a6e771981d3e3975cfcb259f9d1d3bbdc090"}
Jul 27 23:57:49.047108 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.047055    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:57:49.063225 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.063130    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" podStartSLOduration=1.78005332 podStartE2EDuration="2.063109695s" totalImagesPullingTime="283.056375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:49.062901131 +0000 UTC m=+2494.014244544" watchObservedRunningTime="2026-07-27 23:57:49.063109695 +0000 UTC m=+2494.014453082"
Jul 27 23:57:49.089664 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.089610    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n" podStartSLOduration=1.089595178 podStartE2EDuration="1.089595178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:49.088546445 +0000 UTC m=+2494.039889859" watchObservedRunningTime="2026-07-27 23:57:49.089595178 +0000 UTC m=+2494.040938565"
Jul 27 23:57:49.104491 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:49.104444    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6" podStartSLOduration=1.10442675 podStartE2EDuration="1.10442675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:57:49.103462252 +0000 UTC m=+2494.054805677" watchObservedRunningTime="2026-07-27 23:57:49.10442675 +0000 UTC m=+2494.055770136"
Jul 27 23:57:52.009033 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:52.008994    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:57:52.009517 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:52.009347    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:57:54.021556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:54.021500    2570 patch_prober.go:28] interesting pod/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 23:57:54.022057 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:54.021572    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 23:57:55.058655 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.058625    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:57:55.097057 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.097022    2570 generic.go:356] "Generic (PLEG): container finished" podID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerID="429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107" exitCode=0
Jul 27 23:57:55.097227 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.097101    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" event={"ID":"c228a866-c04b-4c21-802c-5b409e6e1b64","Type":"ContainerDied","Data":"429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"}
Jul 27 23:57:55.097629 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.097603    2570 scope.go:117] "RemoveContainer" containerID="429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"
Jul 27 23:57:55.098833 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.098801    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:57:55.098936 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.098895    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:57:55.099014 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:55.098931    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:57:56.105311 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:56.105274    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" event={"ID":"c228a866-c04b-4c21-802c-5b409e6e1b64","Type":"ContainerStarted","Data":"0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa"}
Jul 27 23:57:57.072730 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:57.072693    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:57:57.073413 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:57.073042    2570 patch_prober.go:28] interesting pod/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 23:57:57.073413 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:57.073115    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 23:57:58.073773 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:58.073734    2570 patch_prober.go:28] interesting pod/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 23:57:58.074414 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:58.074380    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 23:57:59.073717 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:59.073669    2570 patch_prober.go:28] interesting pod/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 23:57:59.073942 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:57:59.073758    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 23:58:00.056287 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:00.056252    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:58:00.056920 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:00.056897    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:58:00.072882 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:00.072841    2570 patch_prober.go:28] interesting pod/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 23:58:00.073088 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:00.072904    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 23:58:01.077129 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:01.077091    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:58:01.077627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:01.077455    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:58:01.078206 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:01.078185    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:58:15.307991 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.307890    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"]
Jul 27 23:58:15.308611 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.308132    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" podUID="af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" containerName="maas-api" containerID="cri-o://95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf" gracePeriod=30
Jul 27 23:58:15.361651 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.361617    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"]
Jul 27 23:58:15.361901 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.361871    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7" podUID="764296fd-fc01-4e14-9d72-300bc3708382" containerName="payload-processing" containerID="cri-o://a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e" gracePeriod=30
Jul 27 23:58:15.366388 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.366331    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"]
Jul 27 23:58:15.366669 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.366622    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp" podUID="27a6a5b6-d01b-4dcd-a670-39f486fcb993" containerName="payload-pre-processing" containerID="cri-o://4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d" gracePeriod=30
Jul 27 23:58:15.634175 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.634144    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:58:15.670523 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.670442    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:58:15.680176 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.680149    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:58:15.715523 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715485    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls\") pod \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") "
Jul 27 23:58:15.715719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715581    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6kpm\" (UniqueName: \"kubernetes.io/projected/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-kube-api-access-w6kpm\") pod \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") "
Jul 27 23:58:15.715719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715613    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjfg8\" (UniqueName: \"kubernetes.io/projected/764296fd-fc01-4e14-9d72-300bc3708382-kube-api-access-fjfg8\") pod \"764296fd-fc01-4e14-9d72-300bc3708382\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") "
Jul 27 23:58:15.715719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715655    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764296fd-fc01-4e14-9d72-300bc3708382-plugins-config-volume\") pod \"764296fd-fc01-4e14-9d72-300bc3708382\" (UID: \"764296fd-fc01-4e14-9d72-300bc3708382\") "
Jul 27 23:58:15.715719 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715683    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-openshift-service-ca\") pod \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\" (UID: \"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8\") "
Jul 27 23:58:15.715952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715736    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a6a5b6-d01b-4dcd-a670-39f486fcb993-plugins-config-volume\") pod \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") "
Jul 27 23:58:15.715952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.715777    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpk45\" (UniqueName: \"kubernetes.io/projected/27a6a5b6-d01b-4dcd-a670-39f486fcb993-kube-api-access-kpk45\") pod \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\" (UID: \"27a6a5b6-d01b-4dcd-a670-39f486fcb993\") "
Jul 27 23:58:15.716930 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.716188    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" (UID: "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:15.716930 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.716496    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a6a5b6-d01b-4dcd-a670-39f486fcb993-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27a6a5b6-d01b-4dcd-a670-39f486fcb993" (UID: "27a6a5b6-d01b-4dcd-a670-39f486fcb993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:15.717582 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.717518    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764296fd-fc01-4e14-9d72-300bc3708382-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "764296fd-fc01-4e14-9d72-300bc3708382" (UID: "764296fd-fc01-4e14-9d72-300bc3708382"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:15.718567 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.718509    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-kube-api-access-w6kpm" (OuterVolumeSpecName: "kube-api-access-w6kpm") pod "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" (UID: "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8"). InnerVolumeSpecName "kube-api-access-w6kpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:15.718846 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.718811    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a6a5b6-d01b-4dcd-a670-39f486fcb993-kube-api-access-kpk45" (OuterVolumeSpecName: "kube-api-access-kpk45") pod "27a6a5b6-d01b-4dcd-a670-39f486fcb993" (UID: "27a6a5b6-d01b-4dcd-a670-39f486fcb993"). InnerVolumeSpecName "kube-api-access-kpk45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:15.718945 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.718910    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" (UID: "af4dcf1f-a47c-4076-95cd-c4083c8a9dd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:58:15.719395 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.719338    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764296fd-fc01-4e14-9d72-300bc3708382-kube-api-access-fjfg8" (OuterVolumeSpecName: "kube-api-access-fjfg8") pod "764296fd-fc01-4e14-9d72-300bc3708382" (UID: "764296fd-fc01-4e14-9d72-300bc3708382"). InnerVolumeSpecName "kube-api-access-fjfg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:15.816711 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816688    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/764296fd-fc01-4e14-9d72-300bc3708382-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:15.816711 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816712    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:15.816865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816722    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a6a5b6-d01b-4dcd-a670-39f486fcb993-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:15.816865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816731    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpk45\" (UniqueName: \"kubernetes.io/projected/27a6a5b6-d01b-4dcd-a670-39f486fcb993-kube-api-access-kpk45\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:15.816865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816742    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:15.816865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816750    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6kpm\" (UniqueName: \"kubernetes.io/projected/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8-kube-api-access-w6kpm\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:15.816865 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:15.816759    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjfg8\" (UniqueName: \"kubernetes.io/projected/764296fd-fc01-4e14-9d72-300bc3708382-kube-api-access-fjfg8\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:16.210489 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.210451    2570 generic.go:356] "Generic (PLEG): container finished" podID="764296fd-fc01-4e14-9d72-300bc3708382" containerID="a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e" exitCode=0
Jul 27 23:58:16.210689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.210505    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"
Jul 27 23:58:16.210689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.210539    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7" event={"ID":"764296fd-fc01-4e14-9d72-300bc3708382","Type":"ContainerDied","Data":"a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e"}
Jul 27 23:58:16.210689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.210585    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7" event={"ID":"764296fd-fc01-4e14-9d72-300bc3708382","Type":"ContainerDied","Data":"dec90729b63aeeee0cdd0debb4ead6d08ed05e0cc657ab9caf5c2cf5243c5d8c"}
Jul 27 23:58:16.210689 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.210611    2570 scope.go:117] "RemoveContainer" containerID="a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e"
Jul 27 23:58:16.212022 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.211998    2570 generic.go:356] "Generic (PLEG): container finished" podID="27a6a5b6-d01b-4dcd-a670-39f486fcb993" containerID="4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d" exitCode=0
Jul 27 23:58:16.212136 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.212033    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp" event={"ID":"27a6a5b6-d01b-4dcd-a670-39f486fcb993","Type":"ContainerDied","Data":"4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d"}
Jul 27 23:58:16.212136 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.212060    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"
Jul 27 23:58:16.212136 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.212076    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp" event={"ID":"27a6a5b6-d01b-4dcd-a670-39f486fcb993","Type":"ContainerDied","Data":"22640e494bc35a01f557a684fa1ecbe9230a94ed308a3d99ab00afca4e21438e"}
Jul 27 23:58:16.213420 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.213400    2570 generic.go:356] "Generic (PLEG): container finished" podID="af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" containerID="95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf" exitCode=0
Jul 27 23:58:16.213540 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.213423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" event={"ID":"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8","Type":"ContainerDied","Data":"95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf"}
Jul 27 23:58:16.213540 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.213440    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"
Jul 27 23:58:16.213540 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.213467    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl" event={"ID":"af4dcf1f-a47c-4076-95cd-c4083c8a9dd8","Type":"ContainerDied","Data":"d491a8ada828de42c982a9b48e976c7a025e0b1c2f111c6786d992282f2aef7a"}
Jul 27 23:58:16.219855 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.219831    2570 scope.go:117] "RemoveContainer" containerID="a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e"
Jul 27 23:58:16.220169 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:16.220125    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e\": container with ID starting with a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e not found: ID does not exist" containerID="a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e"
Jul 27 23:58:16.220266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.220177    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e"} err="failed to get container status \"a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e\": rpc error: code = NotFound desc = could not find container \"a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e\": container with ID starting with a02544d6afb963ef2b17896f5b4233ea4184c06d30d8d89ee80b128ad7a6a15e not found: ID does not exist"
Jul 27 23:58:16.220266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.220200    2570 scope.go:117] "RemoveContainer" containerID="4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d"
Jul 27 23:58:16.229570 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.229548    2570 scope.go:117] "RemoveContainer" containerID="4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d"
Jul 27 23:58:16.229851 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:16.229832    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d\": container with ID starting with 4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d not found: ID does not exist" containerID="4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d"
Jul 27 23:58:16.229909 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.229857    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d"} err="failed to get container status \"4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d\": rpc error: code = NotFound desc = could not find container \"4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d\": container with ID starting with 4a1010f7f264c32ebce502cdb95ab0faf2004815fe35767354273cbd62fe827d not found: ID does not exist"
Jul 27 23:58:16.229909 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.229874    2570 scope.go:117] "RemoveContainer" containerID="95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf"
Jul 27 23:58:16.238489 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.238466    2570 scope.go:117] "RemoveContainer" containerID="95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf"
Jul 27 23:58:16.238671 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.238653    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"]
Jul 27 23:58:16.238763 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:16.238743    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf\": container with ID starting with 95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf not found: ID does not exist" containerID="95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf"
Jul 27 23:58:16.238808 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.238770    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf"} err="failed to get container status \"95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf\": rpc error: code = NotFound desc = could not find container \"95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf\": container with ID starting with 95fa0740926043ae8bb6bfd0f1f73def3cf236490257c408da99d968d7731adf not found: ID does not exist"
Jul 27 23:58:16.244216 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.244193    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f3414a-7dd7b8476b-qlrm7"]
Jul 27 23:58:16.254307 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.254286    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"]
Jul 27 23:58:16.257570 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.257548    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3414a-767b97ff65-qs4cl"]
Jul 27 23:58:16.268209 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.268182    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"]
Jul 27 23:58:16.271852 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:16.271828    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f3414a-6d7845ddb8-hvtsp"]
Jul 27 23:58:17.522735 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:17.522700    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a6a5b6-d01b-4dcd-a670-39f486fcb993" path="/var/lib/kubelet/pods/27a6a5b6-d01b-4dcd-a670-39f486fcb993/volumes"
Jul 27 23:58:17.523125 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:17.523069    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764296fd-fc01-4e14-9d72-300bc3708382" path="/var/lib/kubelet/pods/764296fd-fc01-4e14-9d72-300bc3708382/volumes"
Jul 27 23:58:17.523427 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:17.523413    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" path="/var/lib/kubelet/pods/af4dcf1f-a47c-4076-95cd-c4083c8a9dd8/volumes"
Jul 27 23:58:25.939023 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:25.938975    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"]
Jul 27 23:58:25.939471 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:25.939304    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" containerID="cri-o://0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa" gracePeriod=30
Jul 27 23:58:31.079276 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.079235    2570 patch_prober.go:28] interesting pod/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 27 23:58:31.079686 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.079308    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 27 23:58:31.198520 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.198493    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:58:31.253151 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253069    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-credential-socket\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253151 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253113    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-data\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253151 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253129    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-certs\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253158    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvlc7\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-kube-api-access-lvlc7\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253185    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-socket\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253213    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-token\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253276    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-podinfo\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253301    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-envoy\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253344    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c228a866-c04b-4c21-802c-5b409e6e1b64-istiod-ca-cert\") pod \"c228a866-c04b-4c21-802c-5b409e6e1b64\" (UID: \"c228a866-c04b-4c21-802c-5b409e6e1b64\") "
Jul 27 23:58:31.253751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253418    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:31.253751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253472    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:31.253751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253727    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.253751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253748    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.253970 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.253781    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c228a866-c04b-4c21-802c-5b409e6e1b64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:31.254095 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.254058    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-data" (OuterVolumeSpecName: "istio-data") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:31.254095 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.254073    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:31.256027 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.255992    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 23:58:31.256127 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.256065    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-token" (OuterVolumeSpecName: "istio-token") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:31.256127 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.256070    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 23:58:31.256335 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.256312    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-kube-api-access-lvlc7" (OuterVolumeSpecName: "kube-api-access-lvlc7") pod "c228a866-c04b-4c21-802c-5b409e6e1b64" (UID: "c228a866-c04b-4c21-802c-5b409e6e1b64"). InnerVolumeSpecName "kube-api-access-lvlc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:31.294531 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.294499    2570 generic.go:356] "Generic (PLEG): container finished" podID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerID="0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa" exitCode=0
Jul 27 23:58:31.294685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.294544    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"
Jul 27 23:58:31.294685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.294596    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" event={"ID":"c228a866-c04b-4c21-802c-5b409e6e1b64","Type":"ContainerDied","Data":"0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa"}
Jul 27 23:58:31.294685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.294651    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd" event={"ID":"c228a866-c04b-4c21-802c-5b409e6e1b64","Type":"ContainerDied","Data":"c1098c08ddaea6ee0e1d0b1a243f86e9cdb503839dc12ecf7d173bbb92cf31a6"}
Jul 27 23:58:31.294685 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.294676    2570 scope.go:117] "RemoveContainer" containerID="0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa"
Jul 27 23:58:31.304044 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.304029    2570 scope.go:117] "RemoveContainer" containerID="429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"
Jul 27 23:58:31.312050 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.312031    2570 scope.go:117] "RemoveContainer" containerID="0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa"
Jul 27 23:58:31.312294 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:31.312272    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa\": container with ID starting with 0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa not found: ID does not exist" containerID="0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa"
Jul 27 23:58:31.312382 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.312307    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa"} err="failed to get container status \"0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa\": rpc error: code = NotFound desc = could not find container \"0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa\": container with ID starting with 0fd95f8d1a51dbc95c6b3135b9e467f2867a3b95e9a485c862b782ef7c7b42aa not found: ID does not exist"
Jul 27 23:58:31.312382 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.312332    2570 scope.go:117] "RemoveContainer" containerID="429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"
Jul 27 23:58:31.312631 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:31.312614    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107\": container with ID starting with 429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107 not found: ID does not exist" containerID="429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"
Jul 27 23:58:31.312669 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.312638    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107"} err="failed to get container status \"429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107\": rpc error: code = NotFound desc = could not find container \"429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107\": container with ID starting with 429c45e875c08448b3b8c80147cb5ee14296e5e74aa04cfae9a85f5529b69107 not found: ID does not exist"
Jul 27 23:58:31.315849 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.315827    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"]
Jul 27 23:58:31.318966 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.318948    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f3414a-openshift-default-548b9dccc4-jn4fd"]
Jul 27 23:58:31.354976 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.354947    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.354976 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.354973    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.355139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.354992    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c228a866-c04b-4c21-802c-5b409e6e1b64-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.355139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.355001    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.355139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.355009    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvlc7\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-kube-api-access-lvlc7\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.355139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.355019    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c228a866-c04b-4c21-802c-5b409e6e1b64-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.355139 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.355027    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c228a866-c04b-4c21-802c-5b409e6e1b64-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:31.522919 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:31.522841    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" path="/var/lib/kubelet/pods/c228a866-c04b-4c21-802c-5b409e6e1b64/volumes"
Jul 27 23:58:41.658422 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.658385    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"]
Jul 27 23:58:41.658870 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.658643    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" podUID="46dcd98c-0bc1-4396-bdf5-75d9f9f28149" containerName="maas-api" containerID="cri-o://b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f" gracePeriod=30
Jul 27 23:58:41.712261 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.712230    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"]
Jul 27 23:58:41.712522 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.712497    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6" podUID="a99d9cef-6b64-44d8-91e5-1eee762a0cba" containerName="payload-processing" containerID="cri-o://dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474" gracePeriod=30
Jul 27 23:58:41.719557 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.719244    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"]
Jul 27 23:58:41.719557 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.719477    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n" podUID="486b7d90-9c80-4408-aa7c-d81cded4155a" containerName="payload-pre-processing" containerID="cri-o://71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31" gracePeriod=30
Jul 27 23:58:41.993162 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:41.993133    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:58:42.000604 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.000580    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:58:42.012120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.012096    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:58:42.048618 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.048589    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls\") pod \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") "
Jul 27 23:58:42.048789 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.048652    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-openshift-service-ca\") pod \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") "
Jul 27 23:58:42.048789 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.048680    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mvcb\" (UniqueName: \"kubernetes.io/projected/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-kube-api-access-8mvcb\") pod \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\" (UID: \"46dcd98c-0bc1-4396-bdf5-75d9f9f28149\") "
Jul 27 23:58:42.049030 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.049002    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46dcd98c-0bc1-4396-bdf5-75d9f9f28149" (UID: "46dcd98c-0bc1-4396-bdf5-75d9f9f28149"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:42.050869 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.050840    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-kube-api-access-8mvcb" (OuterVolumeSpecName: "kube-api-access-8mvcb") pod "46dcd98c-0bc1-4396-bdf5-75d9f9f28149" (UID: "46dcd98c-0bc1-4396-bdf5-75d9f9f28149"). InnerVolumeSpecName "kube-api-access-8mvcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:42.050967 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.050904    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46dcd98c-0bc1-4396-bdf5-75d9f9f28149" (UID: "46dcd98c-0bc1-4396-bdf5-75d9f9f28149"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 23:58:42.149974 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.149917    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2gv8\" (UniqueName: \"kubernetes.io/projected/486b7d90-9c80-4408-aa7c-d81cded4155a-kube-api-access-j2gv8\") pod \"486b7d90-9c80-4408-aa7c-d81cded4155a\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") "
Jul 27 23:58:42.150159 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150017    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486b7d90-9c80-4408-aa7c-d81cded4155a-plugins-config-volume\") pod \"486b7d90-9c80-4408-aa7c-d81cded4155a\" (UID: \"486b7d90-9c80-4408-aa7c-d81cded4155a\") "
Jul 27 23:58:42.150159 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150064    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9vv7\" (UniqueName: \"kubernetes.io/projected/a99d9cef-6b64-44d8-91e5-1eee762a0cba-kube-api-access-j9vv7\") pod \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") "
Jul 27 23:58:42.150159 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150092    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d9cef-6b64-44d8-91e5-1eee762a0cba-plugins-config-volume\") pod \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\" (UID: \"a99d9cef-6b64-44d8-91e5-1eee762a0cba\") "
Jul 27 23:58:42.150306 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150290    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.150343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150314    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.150343 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150329    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvcb\" (UniqueName: \"kubernetes.io/projected/46dcd98c-0bc1-4396-bdf5-75d9f9f28149-kube-api-access-8mvcb\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.150462 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150438    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486b7d90-9c80-4408-aa7c-d81cded4155a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "486b7d90-9c80-4408-aa7c-d81cded4155a" (UID: "486b7d90-9c80-4408-aa7c-d81cded4155a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:42.150533 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.150515    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99d9cef-6b64-44d8-91e5-1eee762a0cba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a99d9cef-6b64-44d8-91e5-1eee762a0cba" (UID: "a99d9cef-6b64-44d8-91e5-1eee762a0cba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 23:58:42.152294 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.152270    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486b7d90-9c80-4408-aa7c-d81cded4155a-kube-api-access-j2gv8" (OuterVolumeSpecName: "kube-api-access-j2gv8") pod "486b7d90-9c80-4408-aa7c-d81cded4155a" (UID: "486b7d90-9c80-4408-aa7c-d81cded4155a"). InnerVolumeSpecName "kube-api-access-j2gv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:42.152371 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.152287    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99d9cef-6b64-44d8-91e5-1eee762a0cba-kube-api-access-j9vv7" (OuterVolumeSpecName: "kube-api-access-j9vv7") pod "a99d9cef-6b64-44d8-91e5-1eee762a0cba" (UID: "a99d9cef-6b64-44d8-91e5-1eee762a0cba"). InnerVolumeSpecName "kube-api-access-j9vv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 23:58:42.251715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.251643    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/486b7d90-9c80-4408-aa7c-d81cded4155a-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.251715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.251674    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vv7\" (UniqueName: \"kubernetes.io/projected/a99d9cef-6b64-44d8-91e5-1eee762a0cba-kube-api-access-j9vv7\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.251715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.251683    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99d9cef-6b64-44d8-91e5-1eee762a0cba-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.251715 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.251693    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2gv8\" (UniqueName: \"kubernetes.io/projected/486b7d90-9c80-4408-aa7c-d81cded4155a-kube-api-access-j2gv8\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 27 23:58:42.353407 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.353345    2570 generic.go:356] "Generic (PLEG): container finished" podID="486b7d90-9c80-4408-aa7c-d81cded4155a" containerID="71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31" exitCode=0
Jul 27 23:58:42.353584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.353424    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"
Jul 27 23:58:42.353584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.353443    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n" event={"ID":"486b7d90-9c80-4408-aa7c-d81cded4155a","Type":"ContainerDied","Data":"71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31"}
Jul 27 23:58:42.353584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.353492    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n" event={"ID":"486b7d90-9c80-4408-aa7c-d81cded4155a","Type":"ContainerDied","Data":"1df69eb4a5b0dff0f54c266b9a40a6e771981d3e3975cfcb259f9d1d3bbdc090"}
Jul 27 23:58:42.353584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.353513    2570 scope.go:117] "RemoveContainer" containerID="71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31"
Jul 27 23:58:42.355260 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.355232    2570 generic.go:356] "Generic (PLEG): container finished" podID="46dcd98c-0bc1-4396-bdf5-75d9f9f28149" containerID="b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f" exitCode=0
Jul 27 23:58:42.355403 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.355267    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"
Jul 27 23:58:42.355403 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.355326    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" event={"ID":"46dcd98c-0bc1-4396-bdf5-75d9f9f28149","Type":"ContainerDied","Data":"b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f"}
Jul 27 23:58:42.355403 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.355396    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54" event={"ID":"46dcd98c-0bc1-4396-bdf5-75d9f9f28149","Type":"ContainerDied","Data":"0b1c2985fdb323356c49e0bb0f2f65ed3309ef05def03748e7298c49a88fca86"}
Jul 27 23:58:42.356804 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.356776    2570 generic.go:356] "Generic (PLEG): container finished" podID="a99d9cef-6b64-44d8-91e5-1eee762a0cba" containerID="dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474" exitCode=0
Jul 27 23:58:42.356922 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.356823    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"
Jul 27 23:58:42.356922 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.356867    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6" event={"ID":"a99d9cef-6b64-44d8-91e5-1eee762a0cba","Type":"ContainerDied","Data":"dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474"}
Jul 27 23:58:42.357035 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.356937    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6" event={"ID":"a99d9cef-6b64-44d8-91e5-1eee762a0cba","Type":"ContainerDied","Data":"bc159e4867474988100c4e9176eab1e0de6a0d22d9f0ae10a7a472e14a2fcf40"}
Jul 27 23:58:42.364342 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.364317    2570 scope.go:117] "RemoveContainer" containerID="71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31"
Jul 27 23:58:42.364707 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:42.364684    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31\": container with ID starting with 71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31 not found: ID does not exist" containerID="71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31"
Jul 27 23:58:42.364853 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.364720    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31"} err="failed to get container status \"71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31\": rpc error: code = NotFound desc = could not find container \"71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31\": container with ID starting with 71d016149cb8c80e7c9602a6e5934bdd506f15b46282fa46654e091742294f31 not found: ID does not exist"
Jul 27 23:58:42.364853 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.364743    2570 scope.go:117] "RemoveContainer" containerID="b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f"
Jul 27 23:58:42.373205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.373187    2570 scope.go:117] "RemoveContainer" containerID="b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f"
Jul 27 23:58:42.373559 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:42.373541    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f\": container with ID starting with b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f not found: ID does not exist" containerID="b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f"
Jul 27 23:58:42.373613 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.373567    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f"} err="failed to get container status \"b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f\": rpc error: code = NotFound desc = could not find container \"b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f\": container with ID starting with b992afbce8e4ab6519c75f12521a54ea7aa0432b8b3fdd31c0d64306a94e373f not found: ID does not exist"
Jul 27 23:58:42.373613 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.373584    2570 scope.go:117] "RemoveContainer" containerID="dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474"
Jul 27 23:58:42.376886 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.376866    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"]
Jul 27 23:58:42.380718 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.380695    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5d597b-797768f448-96b2n"]
Jul 27 23:58:42.384057 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.384042    2570 scope.go:117] "RemoveContainer" containerID="dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474"
Jul 27 23:58:42.384595 ip-10-0-136-102 kubenswrapper[2570]: E0727 23:58:42.384575    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474\": container with ID starting with dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474 not found: ID does not exist" containerID="dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474"
Jul 27 23:58:42.384672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.384603    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474"} err="failed to get container status \"dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474\": rpc error: code = NotFound desc = could not find container \"dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474\": container with ID starting with dca94c2b54038d3517227c5d98b3f6d01826ca538e16e916ab304a2629853474 not found: ID does not exist"
Jul 27 23:58:42.393606 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.393581    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"]
Jul 27 23:58:42.398137 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.398116    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5d597b-6ffc454d99-bcs54"]
Jul 27 23:58:42.409077 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.409057    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"]
Jul 27 23:58:42.412615 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:42.412593    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5d597b-69cb57bf58-gh5g6"]
Jul 27 23:58:43.522406 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:43.522378    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46dcd98c-0bc1-4396-bdf5-75d9f9f28149" path="/var/lib/kubelet/pods/46dcd98c-0bc1-4396-bdf5-75d9f9f28149/volumes"
Jul 27 23:58:43.522780 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:43.522734    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486b7d90-9c80-4408-aa7c-d81cded4155a" path="/var/lib/kubelet/pods/486b7d90-9c80-4408-aa7c-d81cded4155a/volumes"
Jul 27 23:58:43.523062 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:43.523049    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99d9cef-6b64-44d8-91e5-1eee762a0cba" path="/var/lib/kubelet/pods/a99d9cef-6b64-44d8-91e5-1eee762a0cba/volumes"
Jul 27 23:58:53.637259 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637228    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"]
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637607    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46dcd98c-0bc1-4396-bdf5-75d9f9f28149" containerName="maas-api"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637623    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="46dcd98c-0bc1-4396-bdf5-75d9f9f28149" containerName="maas-api"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637643    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486b7d90-9c80-4408-aa7c-d81cded4155a" containerName="payload-pre-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637649    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="486b7d90-9c80-4408-aa7c-d81cded4155a" containerName="payload-pre-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637665    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a6a5b6-d01b-4dcd-a670-39f486fcb993" containerName="payload-pre-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637674    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a6a5b6-d01b-4dcd-a670-39f486fcb993" containerName="payload-pre-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637687    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764296fd-fc01-4e14-9d72-300bc3708382" containerName="payload-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637697    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="764296fd-fc01-4e14-9d72-300bc3708382" containerName="payload-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637708    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637716    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637726    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99d9cef-6b64-44d8-91e5-1eee762a0cba" containerName="payload-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637731    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99d9cef-6b64-44d8-91e5-1eee762a0cba" containerName="payload-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637746    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" containerName="maas-api"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637751    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" containerName="maas-api"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637800    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="46dcd98c-0bc1-4396-bdf5-75d9f9f28149" containerName="maas-api"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637809    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a99d9cef-6b64-44d8-91e5-1eee762a0cba" containerName="payload-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637815    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="486b7d90-9c80-4408-aa7c-d81cded4155a" containerName="payload-pre-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637824    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637830    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a6a5b6-d01b-4dcd-a670-39f486fcb993" containerName="payload-pre-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637835    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4dcf1f-a47c-4076-95cd-c4083c8a9dd8" containerName="maas-api"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637841    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="764296fd-fc01-4e14-9d72-300bc3708382" containerName="payload-processing"
Jul 27 23:58:53.637867 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.637847    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy"
Jul 27 23:58:53.640622 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.640607    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.643211 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.643193    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-42d790-openshift-default-dockercfg-4s64w\""
Jul 27 23:58:53.649751 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.649729    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"]
Jul 27 23:58:53.749479 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749446    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istiod-ca-cert\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749499    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-socket\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-envoy\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-credential-socket\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749635    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llvhb\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-kube-api-access-llvhb\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-certs\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749734    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-podinfo\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749772    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-data\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.749837 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.749800    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-token\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.850943 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.850904    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-socket\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.850952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-envoy\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.850995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-credential-socket\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851022    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llvhb\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-kube-api-access-llvhb\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851053    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-certs\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851084    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-podinfo\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-data\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851162    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-token\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851210    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istiod-ca-cert\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851386    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-socket\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851506 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851393    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-credential-socket\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851768 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851592    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-certs\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.851768 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.851614    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-data\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.852069 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.852039    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istiod-ca-cert\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.854318 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.854295    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-envoy\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.854481 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.854462    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-podinfo\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.860647 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.860621    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-token\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.860774 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.860755    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llvhb\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-kube-api-access-llvhb\") pod \"e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") " pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:53.952198 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:53.952097    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:54.087940 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.087913    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"]
Jul 27 23:58:54.090403 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:58:54.090377    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ca41d0_f50e_4dce_aedd_cbc5c75b8c05.slice/crio-98f13ff5f8c4692c67bb0e0241244873bcfb99a6465056107dd39a229a28c1e4 WatchSource:0}: Error finding container 98f13ff5f8c4692c67bb0e0241244873bcfb99a6465056107dd39a229a28c1e4: Status 404 returned error can't find the container with id 98f13ff5f8c4692c67bb0e0241244873bcfb99a6465056107dd39a229a28c1e4
Jul 27 23:58:54.093018 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.092981    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:58:54.093100 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.093050    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:58:54.093100 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.093079    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:58:54.422190 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.422155    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" event={"ID":"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05","Type":"ContainerStarted","Data":"0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"}
Jul 27 23:58:54.422402 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.422226    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" event={"ID":"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05","Type":"ContainerStarted","Data":"98f13ff5f8c4692c67bb0e0241244873bcfb99a6465056107dd39a229a28c1e4"}
Jul 27 23:58:54.443258 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.443203    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podStartSLOduration=1.44318772 podStartE2EDuration="1.44318772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:58:54.440007819 +0000 UTC m=+2559.391351209" watchObservedRunningTime="2026-07-27 23:58:54.44318772 +0000 UTC m=+2559.394531107"
Jul 27 23:58:54.952886 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:54.952851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:55.957348 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:55.957313    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:56.433133 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.433102    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:56.434158 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.434137    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:58:56.501708 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.501676    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"]
Jul 27 23:58:56.502076 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.502061    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy"
Jul 27 23:58:56.502076 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.502076    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c228a866-c04b-4c21-802c-5b409e6e1b64" containerName="istio-proxy"
Jul 27 23:58:56.505336 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.505316    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.508210 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.508189    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-42d790-serving-cert\""
Jul 27 23:58:56.522108 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.522076    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"]
Jul 27 23:58:56.680941 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.680904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghgr\" (UniqueName: \"kubernetes.io/projected/a6abc96b-6ad3-4d5b-9add-38349c43bd26-kube-api-access-kghgr\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.681126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.680950    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6abc96b-6ad3-4d5b-9add-38349c43bd26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.681126 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.681018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6abc96b-6ad3-4d5b-9add-38349c43bd26-maas-api-tls\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.782341 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.782246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghgr\" (UniqueName: \"kubernetes.io/projected/a6abc96b-6ad3-4d5b-9add-38349c43bd26-kube-api-access-kghgr\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.782341 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.782295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6abc96b-6ad3-4d5b-9add-38349c43bd26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.782599 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.782342    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6abc96b-6ad3-4d5b-9add-38349c43bd26-maas-api-tls\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.783066 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.783038    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6abc96b-6ad3-4d5b-9add-38349c43bd26-openshift-service-ca\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.784985 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.784968    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6abc96b-6ad3-4d5b-9add-38349c43bd26-maas-api-tls\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.790896 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.790876    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghgr\" (UniqueName: \"kubernetes.io/projected/a6abc96b-6ad3-4d5b-9add-38349c43bd26-kube-api-access-kghgr\") pod \"maas-api-e2e-shared-a-42d790-78df554c4f-n67gz\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.817120 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.817090    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:56.955986 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:56.955961    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"]
Jul 27 23:58:56.957999 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:58:56.957975    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6abc96b_6ad3_4d5b_9add_38349c43bd26.slice/crio-3e98e6f000b1e5ee7c261fefc5580710f7d43e57def4b750a551b7c5637bfae6 WatchSource:0}: Error finding container 3e98e6f000b1e5ee7c261fefc5580710f7d43e57def4b750a551b7c5637bfae6: Status 404 returned error can't find the container with id 3e98e6f000b1e5ee7c261fefc5580710f7d43e57def4b750a551b7c5637bfae6
Jul 27 23:58:57.166650 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.166611    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"]
Jul 27 23:58:57.171287 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.171272    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.174215 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.174190    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-42d790\""
Jul 27 23:58:57.174340 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.174251    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-42d790-dockercfg-tqtgw\""
Jul 27 23:58:57.180982 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.180954    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"]
Jul 27 23:58:57.196431 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.196397    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"]
Jul 27 23:58:57.200617 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.200597    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.207280 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.207255    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"]
Jul 27 23:58:57.287228 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.287203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psl5\" (UniqueName: \"kubernetes.io/projected/00e2931b-e84b-46bd-91f2-f52ff4697b31-kube-api-access-5psl5\") pod \"payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.287379 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.287289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c481186c-38f5-4aa7-a502-bc4f0a652044-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") " pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.287379 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.287331    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j5h\" (UniqueName: \"kubernetes.io/projected/c481186c-38f5-4aa7-a502-bc4f0a652044-kube-api-access-g7j5h\") pod \"payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") " pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.287479 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.287384    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e2931b-e84b-46bd-91f2-f52ff4697b31-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.388246 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.388207    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c481186c-38f5-4aa7-a502-bc4f0a652044-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") " pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.388443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.388252    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j5h\" (UniqueName: \"kubernetes.io/projected/c481186c-38f5-4aa7-a502-bc4f0a652044-kube-api-access-g7j5h\") pod \"payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") " pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.388443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.388278    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e2931b-e84b-46bd-91f2-f52ff4697b31-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.388443 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.388327    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psl5\" (UniqueName: \"kubernetes.io/projected/00e2931b-e84b-46bd-91f2-f52ff4697b31-kube-api-access-5psl5\") pod \"payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.388962 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.388942    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c481186c-38f5-4aa7-a502-bc4f0a652044-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") " pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.389034 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.389017    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e2931b-e84b-46bd-91f2-f52ff4697b31-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.395881 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.395854    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j5h\" (UniqueName: \"kubernetes.io/projected/c481186c-38f5-4aa7-a502-bc4f0a652044-kube-api-access-g7j5h\") pod \"payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") " pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.396007 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.395887    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psl5\" (UniqueName: \"kubernetes.io/projected/00e2931b-e84b-46bd-91f2-f52ff4697b31-kube-api-access-5psl5\") pod \"payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.440131 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.440038    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz" event={"ID":"a6abc96b-6ad3-4d5b-9add-38349c43bd26","Type":"ContainerStarted","Data":"08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70"}
Jul 27 23:58:57.440131 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.440091    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz" event={"ID":"a6abc96b-6ad3-4d5b-9add-38349c43bd26","Type":"ContainerStarted","Data":"3e98e6f000b1e5ee7c261fefc5580710f7d43e57def4b750a551b7c5637bfae6"}
Jul 27 23:58:57.440329 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.440294    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:58:57.462627 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.462570    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz" podStartSLOduration=1.137511303 podStartE2EDuration="1.462553039s" totalImagesPullingTime="325.041736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:58:57.462106114 +0000 UTC m=+2562.413449521" watchObservedRunningTime="2026-07-27 23:58:57.462553039 +0000 UTC m=+2562.413896427"
Jul 27 23:58:57.482773 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.482731    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:57.513213 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.513185    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:57.666760 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.666668    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"]
Jul 27 23:58:57.670506 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:58:57.670470    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e2931b_e84b_46bd_91f2_f52ff4697b31.slice/crio-f0f97996fad111e8b42872061f89733bb5cedf8d618e7517b9ea13dd51c51faa WatchSource:0}: Error finding container f0f97996fad111e8b42872061f89733bb5cedf8d618e7517b9ea13dd51c51faa: Status 404 returned error can't find the container with id f0f97996fad111e8b42872061f89733bb5cedf8d618e7517b9ea13dd51c51faa
Jul 27 23:58:57.705907 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:57.705881    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"]
Jul 27 23:58:57.707764 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:58:57.707723    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc481186c_38f5_4aa7_a502_bc4f0a652044.slice/crio-5309e4610de1cec5c69f0de7d7b9b8b454195b18d166f8bea19accc8420786b9 WatchSource:0}: Error finding container 5309e4610de1cec5c69f0de7d7b9b8b454195b18d166f8bea19accc8420786b9: Status 404 returned error can't find the container with id 5309e4610de1cec5c69f0de7d7b9b8b454195b18d166f8bea19accc8420786b9
Jul 27 23:58:58.446769 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.446728    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm" event={"ID":"c481186c-38f5-4aa7-a502-bc4f0a652044","Type":"ContainerStarted","Data":"73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc"}
Jul 27 23:58:58.447205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.446806    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:58:58.447205 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.446837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm" event={"ID":"c481186c-38f5-4aa7-a502-bc4f0a652044","Type":"ContainerStarted","Data":"5309e4610de1cec5c69f0de7d7b9b8b454195b18d166f8bea19accc8420786b9"}
Jul 27 23:58:58.448213 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.448184    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr" event={"ID":"00e2931b-e84b-46bd-91f2-f52ff4697b31","Type":"ContainerStarted","Data":"a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0"}
Jul 27 23:58:58.448330 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.448235    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr" event={"ID":"00e2931b-e84b-46bd-91f2-f52ff4697b31","Type":"ContainerStarted","Data":"f0f97996fad111e8b42872061f89733bb5cedf8d618e7517b9ea13dd51c51faa"}
Jul 27 23:58:58.448330 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.448293    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:58:58.464498 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.464454    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm" podStartSLOduration=1.464439318 podStartE2EDuration="1.464439318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:58:58.461712056 +0000 UTC m=+2563.413055442" watchObservedRunningTime="2026-07-27 23:58:58.464439318 +0000 UTC m=+2563.415782683"
Jul 27 23:58:58.480691 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:58:58.480638    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr" podStartSLOduration=1.480619279 podStartE2EDuration="1.480619279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:58:58.478160176 +0000 UTC m=+2563.429503564" watchObservedRunningTime="2026-07-27 23:58:58.480619279 +0000 UTC m=+2563.431962669"
Jul 27 23:59:03.453905 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:03.453873    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 27 23:59:04.786615 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.786579    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"]
Jul 27 23:59:04.789876 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.789851    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.792547 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.792525    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4a8122-serving-cert\""
Jul 27 23:59:04.802553 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.802525    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"]
Jul 27 23:59:04.858921 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.858885    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2c0064-2eff-4db3-944a-8f11f84ae59c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.859094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.858926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ksw\" (UniqueName: \"kubernetes.io/projected/bf2c0064-2eff-4db3-944a-8f11f84ae59c-kube-api-access-d6ksw\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.859094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.859012    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2c0064-2eff-4db3-944a-8f11f84ae59c-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.959670 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.959635    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2c0064-2eff-4db3-944a-8f11f84ae59c-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.959856 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.959721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2c0064-2eff-4db3-944a-8f11f84ae59c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.959856 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.959759    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ksw\" (UniqueName: \"kubernetes.io/projected/bf2c0064-2eff-4db3-944a-8f11f84ae59c-kube-api-access-d6ksw\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.960507 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.960484    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2c0064-2eff-4db3-944a-8f11f84ae59c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.962390 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.962372    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2c0064-2eff-4db3-944a-8f11f84ae59c-maas-api-tls\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:04.968083 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:04.968055    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ksw\" (UniqueName: \"kubernetes.io/projected/bf2c0064-2eff-4db3-944a-8f11f84ae59c-kube-api-access-d6ksw\") pod \"maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:05.104170 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.104081    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:05.244665 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.244286    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"]
Jul 27 23:59:05.247553 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:59:05.247519    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2c0064_2eff_4db3_944a_8f11f84ae59c.slice/crio-de6bdaaa5cbff3f10e04d534f2c452955036d2858a7540141e624d3ae4539df8 WatchSource:0}: Error finding container de6bdaaa5cbff3f10e04d534f2c452955036d2858a7540141e624d3ae4539df8: Status 404 returned error can't find the container with id de6bdaaa5cbff3f10e04d534f2c452955036d2858a7540141e624d3ae4539df8
Jul 27 23:59:05.376094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.376013    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"]
Jul 27 23:59:05.380952 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.380933    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.383872 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.383850    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-4a8122\""
Jul 27 23:59:05.384017 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.384000    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-4a8122-dockercfg-lxbmd\""
Jul 27 23:59:05.389237 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.388996    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"]
Jul 27 23:59:05.412584 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.412552    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"]
Jul 27 23:59:05.416826 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.416805    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.423674 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.423647    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"]
Jul 27 23:59:05.463625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.463594    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts2nt\" (UniqueName: \"kubernetes.io/projected/e523ad06-3470-421f-b533-0b023e108cb4-kube-api-access-ts2nt\") pod \"payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.463625 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.463632    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74sjl\" (UniqueName: \"kubernetes.io/projected/4b12240f-44aa-48a2-bad9-8f7fed96987b-kube-api-access-74sjl\") pod \"payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.463866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.463691    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b12240f-44aa-48a2-bad9-8f7fed96987b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.463866 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.463733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e523ad06-3470-421f-b533-0b023e108cb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.491094 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.491047    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q" event={"ID":"bf2c0064-2eff-4db3-944a-8f11f84ae59c","Type":"ContainerStarted","Data":"de6bdaaa5cbff3f10e04d534f2c452955036d2858a7540141e624d3ae4539df8"}
Jul 27 23:59:05.564436 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.564396    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b12240f-44aa-48a2-bad9-8f7fed96987b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.564651 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.564457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e523ad06-3470-421f-b533-0b023e108cb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.564651 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.564527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts2nt\" (UniqueName: \"kubernetes.io/projected/e523ad06-3470-421f-b533-0b023e108cb4-kube-api-access-ts2nt\") pod \"payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.564651 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.564560    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74sjl\" (UniqueName: \"kubernetes.io/projected/4b12240f-44aa-48a2-bad9-8f7fed96987b-kube-api-access-74sjl\") pod \"payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.565266 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.565242    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e523ad06-3470-421f-b533-0b023e108cb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.565335 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.565311    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b12240f-44aa-48a2-bad9-8f7fed96987b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.572944 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.572912    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74sjl\" (UniqueName: \"kubernetes.io/projected/4b12240f-44aa-48a2-bad9-8f7fed96987b-kube-api-access-74sjl\") pod \"payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.573048 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.572960    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts2nt\" (UniqueName: \"kubernetes.io/projected/e523ad06-3470-421f-b533-0b023e108cb4-kube-api-access-ts2nt\") pod \"payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.693885 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.693860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:05.729228 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.729197    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:05.860703 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:05.860664    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"]
Jul 27 23:59:06.092060 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.092024    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"]
Jul 27 23:59:06.095320 ip-10-0-136-102 kubenswrapper[2570]: W0727 23:59:06.095294    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b12240f_44aa_48a2_bad9_8f7fed96987b.slice/crio-58ee9ed3cee3b5365c6d83a74f14af4051a2735bff490f0b16f460f45f3c5e7b WatchSource:0}: Error finding container 58ee9ed3cee3b5365c6d83a74f14af4051a2735bff490f0b16f460f45f3c5e7b: Status 404 returned error can't find the container with id 58ee9ed3cee3b5365c6d83a74f14af4051a2735bff490f0b16f460f45f3c5e7b
Jul 27 23:59:06.501512 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.501449    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt" event={"ID":"e523ad06-3470-421f-b533-0b023e108cb4","Type":"ContainerStarted","Data":"94d9c31d8991bae2c399af7363eaee25da333a6ac015d2aa3b33feee13dd77f1"}
Jul 27 23:59:06.501745 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.501546    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt" event={"ID":"e523ad06-3470-421f-b533-0b023e108cb4","Type":"ContainerStarted","Data":"e6e9f9c34a6cc56de10a4bdaaaab9d308c833cf34299d4083e2494d25cab93e9"}
Jul 27 23:59:06.502530 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.502506    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:06.509746 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.508986    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q" event={"ID":"bf2c0064-2eff-4db3-944a-8f11f84ae59c","Type":"ContainerStarted","Data":"8f1fdd2941e20f9d66652fcd1c5a3f5932235fb43074bcbda9fa634cbffa569e"}
Jul 27 23:59:06.509746 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.509684    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:06.512323 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.512296    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6" event={"ID":"4b12240f-44aa-48a2-bad9-8f7fed96987b","Type":"ContainerStarted","Data":"7939e2cddccb0264d5538c4e2be5445f5b3ef388d8514d13a4ede87f4690e96d"}
Jul 27 23:59:06.512483 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.512344    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6" event={"ID":"4b12240f-44aa-48a2-bad9-8f7fed96987b","Type":"ContainerStarted","Data":"58ee9ed3cee3b5365c6d83a74f14af4051a2735bff490f0b16f460f45f3c5e7b"}
Jul 27 23:59:06.512556 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.512500    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:06.517150 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.517096    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt" podStartSLOduration=1.517080933 podStartE2EDuration="1.517080933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:06.516834639 +0000 UTC m=+2571.468178049" watchObservedRunningTime="2026-07-27 23:59:06.517080933 +0000 UTC m=+2571.468424320"
Jul 27 23:59:06.533085 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.533039    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q" podStartSLOduration=2.111079873 podStartE2EDuration="2.53302503s" totalImagesPullingTime="421.945157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:06.531268714 +0000 UTC m=+2571.482612123" watchObservedRunningTime="2026-07-27 23:59:06.53302503 +0000 UTC m=+2571.484368418"
Jul 27 23:59:06.547305 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:06.547259    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6" podStartSLOduration=1.5472441030000001 podStartE2EDuration="1.547244103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:06.545950365 +0000 UTC m=+2571.497293777" watchObservedRunningTime="2026-07-27 23:59:06.547244103 +0000 UTC m=+2571.498587516"
Jul 27 23:59:09.457607 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:09.457570    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 27 23:59:09.458403 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:09.458376    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 27 23:59:12.474560 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:12.474508    2570 patch_prober.go:28] interesting pod/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 23:59:12.474945 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:12.474599    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 23:59:13.528142 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.528111    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 27 23:59:13.549504 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.549475    2570 generic.go:356] "Generic (PLEG): container finished" podID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerID="0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd" exitCode=0
Jul 27 23:59:13.549669 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.549551    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" event={"ID":"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05","Type":"ContainerDied","Data":"0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"}
Jul 27 23:59:13.550093 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.550069    2570 scope.go:117] "RemoveContainer" containerID="0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"
Jul 27 23:59:13.551240 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.551208    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:59:13.551367 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.551297    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:59:13.551367 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:13.551335    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 23:59:14.556921 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:14.556886    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" event={"ID":"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05","Type":"ContainerStarted","Data":"f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554"}
Jul 27 23:59:14.953423 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:14.953385    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:59:14.953626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:14.953513    2570 patch_prober.go:28] interesting pod/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 27 23:59:14.953626 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:14.953557    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 27 23:59:15.953672 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:15.953629    2570 patch_prober.go:28] interesting pod/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 27 23:59:15.954194 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:15.953701    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 27 23:59:16.955805 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:16.955767    2570 patch_prober.go:28] interesting pod/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 27 23:59:16.956386 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:16.956331    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 27 23:59:17.522997 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:17.522956    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 27 23:59:17.952577 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:17.952540    2570 patch_prober.go:28] interesting pod/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Jul 27 23:59:17.952860 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:17.952831    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Jul 27 23:59:18.525548 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:18.525517    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 27 23:59:18.956017 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:18.955977    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:59:18.956401 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:18.956380    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 27 23:59:18.957041 ip-10-0-136-102 kubenswrapper[2570]: I0727 23:59:18.957025    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 28 00:01:15.612048 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:01:15.611976    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:01:15.616876 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:01:15.616855    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:06:15.650470 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:06:15.650439    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:06:15.653396 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:06:15.653368    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:08:25.911478 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:25.911444    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"]
Jul 28 00:08:25.915233 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:25.915210    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:25.917902 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:25.917874    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4a6d94-serving-cert\""
Jul 28 00:08:25.924724 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:25.923885    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"]
Jul 28 00:08:26.036648 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.036603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/c621b735-7691-4086-9cdc-d6ddd62c7782-kube-api-access-bgqrz\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.036851 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.036665    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c621b735-7691-4086-9cdc-d6ddd62c7782-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.036851 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.036746    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c621b735-7691-4086-9cdc-d6ddd62c7782-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.137980 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.137942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/c621b735-7691-4086-9cdc-d6ddd62c7782-kube-api-access-bgqrz\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.138179 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.137998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c621b735-7691-4086-9cdc-d6ddd62c7782-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.138179 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.138029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c621b735-7691-4086-9cdc-d6ddd62c7782-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.138675 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.138651    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c621b735-7691-4086-9cdc-d6ddd62c7782-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.140866 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.140836    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c621b735-7691-4086-9cdc-d6ddd62c7782-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.145958 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.145938    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/c621b735-7691-4086-9cdc-d6ddd62c7782-kube-api-access-bgqrz\") pod \"maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.228666 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.228576    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:26.385693 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.385669    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"]
Jul 28 00:08:26.387825 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:08:26.387791    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc621b735_7691_4086_9cdc_d6ddd62c7782.slice/crio-7e65ff5e9fe3473078961b566752e6bee7739d1cdb6b2117c4db70dc3b5c04dd WatchSource:0}: Error finding container 7e65ff5e9fe3473078961b566752e6bee7739d1cdb6b2117c4db70dc3b5c04dd: Status 404 returned error can't find the container with id 7e65ff5e9fe3473078961b566752e6bee7739d1cdb6b2117c4db70dc3b5c04dd
Jul 28 00:08:26.389476 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.389452    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 00:08:26.585795 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.585705    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"]
Jul 28 00:08:26.590384 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.590342    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.594034 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.594010    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4a6d94-dockercfg-dgg4h\""
Jul 28 00:08:26.594183 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.594013    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4a6d94\""
Jul 28 00:08:26.604335 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.604307    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"]
Jul 28 00:08:26.744172 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.744134    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6fr\" (UniqueName: \"kubernetes.io/projected/bf035b6c-ae03-4654-b111-bb3db4d51af5-kube-api-access-jl6fr\") pod \"payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.744366 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.744180    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf035b6c-ae03-4654-b111-bb3db4d51af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.845418 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.845289    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6fr\" (UniqueName: \"kubernetes.io/projected/bf035b6c-ae03-4654-b111-bb3db4d51af5-kube-api-access-jl6fr\") pod \"payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.845418 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.845341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf035b6c-ae03-4654-b111-bb3db4d51af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.846142 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.846100    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf035b6c-ae03-4654-b111-bb3db4d51af5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.853884 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.853858    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6fr\" (UniqueName: \"kubernetes.io/projected/bf035b6c-ae03-4654-b111-bb3db4d51af5-kube-api-access-jl6fr\") pod \"payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:26.901482 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:26.901444    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:27.046496 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:08:27.046469    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf035b6c_ae03_4654_b111_bb3db4d51af5.slice/crio-0824c2a66e18c3b016c3bd4b66cf60efe3feb2747f182280b6bfeb616552148e WatchSource:0}: Error finding container 0824c2a66e18c3b016c3bd4b66cf60efe3feb2747f182280b6bfeb616552148e: Status 404 returned error can't find the container with id 0824c2a66e18c3b016c3bd4b66cf60efe3feb2747f182280b6bfeb616552148e
Jul 28 00:08:27.046804 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.046549    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"]
Jul 28 00:08:27.351269 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.351159    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v" event={"ID":"c621b735-7691-4086-9cdc-d6ddd62c7782","Type":"ContainerStarted","Data":"fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85"}
Jul 28 00:08:27.351468 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.351302    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:27.351468 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.351345    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v" event={"ID":"c621b735-7691-4086-9cdc-d6ddd62c7782","Type":"ContainerStarted","Data":"7e65ff5e9fe3473078961b566752e6bee7739d1cdb6b2117c4db70dc3b5c04dd"}
Jul 28 00:08:27.352970 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.352931    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd" event={"ID":"bf035b6c-ae03-4654-b111-bb3db4d51af5","Type":"ContainerStarted","Data":"1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97"}
Jul 28 00:08:27.353099 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.352979    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd" event={"ID":"bf035b6c-ae03-4654-b111-bb3db4d51af5","Type":"ContainerStarted","Data":"0824c2a66e18c3b016c3bd4b66cf60efe3feb2747f182280b6bfeb616552148e"}
Jul 28 00:08:27.353099 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.353028    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:08:27.370560 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.370504    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v" podStartSLOduration=1.769022107 podStartE2EDuration="2.370480391s" totalImagesPullingTime="601.458284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:27.369115795 +0000 UTC m=+3132.320459194" watchObservedRunningTime="2026-07-28 00:08:27.370480391 +0000 UTC m=+3132.321823778"
Jul 28 00:08:27.384123 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:27.384076    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd" podStartSLOduration=1.384060209 podStartE2EDuration="1.384060209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:27.383312589 +0000 UTC m=+3132.334656004" watchObservedRunningTime="2026-07-28 00:08:27.384060209 +0000 UTC m=+3132.335403595"
Jul 28 00:08:33.367177 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:33.367142    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:08:38.362584 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:08:38.362494    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:09:11.337214 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:09:11.337185    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mcpnn_b0d6a3ab-abed-40dd-a462-a521cbdef40e/payload-processing/0.log"
Jul 28 00:10:03.408840 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:03.408806    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mcpnn_b0d6a3ab-abed-40dd-a462-a521cbdef40e/payload-processing/0.log"
Jul 28 00:10:04.049248 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.049217    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"]
Jul 28 00:10:04.052932 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.052912    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.055730 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.055713    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a35fc9-openshift-default-dockercfg-mxgdb\""
Jul 28 00:10:04.065557 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.065534    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"]
Jul 28 00:10:04.076819 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.076788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-socket\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.076921 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.076826    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-istio-token\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.076921 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.076854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e161adfa-1131-4054-884e-720b606cca79-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.077019 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.076919    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-certs\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.077107 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.077084    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e161adfa-1131-4054-884e-720b606cca79-istio-podinfo\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.077150 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.077132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-credential-socket\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.077203 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.077187    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-envoy\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.077272 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.077250    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-data\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.077324 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.077281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxfn\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-kube-api-access-ptxfn\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178244 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178213    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e161adfa-1131-4054-884e-720b606cca79-istio-podinfo\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178460 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178256    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-credential-socket\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178460 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178289    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-envoy\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178460 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-data\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178499    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxfn\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-kube-api-access-ptxfn\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-socket\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178637    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-istio-token\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178663    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e161adfa-1131-4054-884e-720b606cca79-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178693    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-certs\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178992 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178792    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-credential-socket\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.178992 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178836    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-data\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.179118 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.178988    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-socket\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.179118 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.179053    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-certs\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.179486 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.179429    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e161adfa-1131-4054-884e-720b606cca79-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.181247 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.181225    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-envoy\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.181592 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.181570    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e161adfa-1131-4054-884e-720b606cca79-istio-podinfo\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.186343 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.186318    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-istio-token\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.186473 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.186455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxfn\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-kube-api-access-ptxfn\") pod \"e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") " pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.364328 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.364241    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:04.508593 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.508564    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"]
Jul 28 00:10:04.510007 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:10:04.509968    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode161adfa_1131_4054_884e_720b606cca79.slice/crio-fcc2d76eab896e04aac6eef96ef3fd6f603a53104ab4e766d1c61bba3863e6db WatchSource:0}: Error finding container fcc2d76eab896e04aac6eef96ef3fd6f603a53104ab4e766d1c61bba3863e6db: Status 404 returned error can't find the container with id fcc2d76eab896e04aac6eef96ef3fd6f603a53104ab4e766d1c61bba3863e6db
Jul 28 00:10:04.512333 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.512302    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 00:10:04.512437 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.512384    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 00:10:04.512437 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.512416    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 00:10:04.856054 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.856017    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" event={"ID":"e161adfa-1131-4054-884e-720b606cca79","Type":"ContainerStarted","Data":"4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"}
Jul 28 00:10:04.856245 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.856077    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" event={"ID":"e161adfa-1131-4054-884e-720b606cca79","Type":"ContainerStarted","Data":"fcc2d76eab896e04aac6eef96ef3fd6f603a53104ab4e766d1c61bba3863e6db"}
Jul 28 00:10:04.898719 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:04.898651    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" podStartSLOduration=0.898631194 podStartE2EDuration="898.631194ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:10:04.897306407 +0000 UTC m=+3229.848649813" watchObservedRunningTime="2026-07-28 00:10:04.898631194 +0000 UTC m=+3229.849974581"
Jul 28 00:10:05.364757 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:05.364717    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:05.370372 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:05.370332    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:05.861895 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:05.861810    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:05.862862 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:05.862844    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:07.670916 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.670875    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"]
Jul 28 00:10:07.674705 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.674684    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.677385 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.677338    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a35fc9\""
Jul 28 00:10:07.677470 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.677456    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a35fc9-dockercfg-btx8t\""
Jul 28 00:10:07.688290 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.688259    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"]
Jul 28 00:10:07.704999 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.704963    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"]
Jul 28 00:10:07.710952 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.710917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2nj\" (UniqueName: \"kubernetes.io/projected/0838d5f2-dc6b-4ff8-87b4-86637535be7c-kube-api-access-hs2nj\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.711094 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.710983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0838d5f2-dc6b-4ff8-87b4-86637535be7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.714668 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.714649    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.720611 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.720584    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"]
Jul 28 00:10:07.811717 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.811674    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac73c56-6096-44d1-9191-7f402995d6b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.811937 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.811735    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2nj\" (UniqueName: \"kubernetes.io/projected/0838d5f2-dc6b-4ff8-87b4-86637535be7c-kube-api-access-hs2nj\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.811937 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.811807    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0838d5f2-dc6b-4ff8-87b4-86637535be7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.811937 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.811876    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfr7\" (UniqueName: \"kubernetes.io/projected/4ac73c56-6096-44d1-9191-7f402995d6b8-kube-api-access-smfr7\") pod \"payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.812507 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.812487    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0838d5f2-dc6b-4ff8-87b4-86637535be7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.819760 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.819731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2nj\" (UniqueName: \"kubernetes.io/projected/0838d5f2-dc6b-4ff8-87b4-86637535be7c-kube-api-access-hs2nj\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:07.912988 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.912937    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfr7\" (UniqueName: \"kubernetes.io/projected/4ac73c56-6096-44d1-9191-7f402995d6b8-kube-api-access-smfr7\") pod \"payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.913196 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.913066    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac73c56-6096-44d1-9191-7f402995d6b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.913729 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.913707    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac73c56-6096-44d1-9191-7f402995d6b8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.921181 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.921123    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfr7\" (UniqueName: \"kubernetes.io/projected/4ac73c56-6096-44d1-9191-7f402995d6b8-kube-api-access-smfr7\") pod \"payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:07.985898 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:07.985857    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:08.026596 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.026566    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:08.138548 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.138492    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"]
Jul 28 00:10:08.142857 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:10:08.142827    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0838d5f2_dc6b_4ff8_87b4_86637535be7c.slice/crio-c82951e780ef063080c21911c61807b0b28bc1f5ac1395f2b1cfb098b3f1ae89 WatchSource:0}: Error finding container c82951e780ef063080c21911c61807b0b28bc1f5ac1395f2b1cfb098b3f1ae89: Status 404 returned error can't find the container with id c82951e780ef063080c21911c61807b0b28bc1f5ac1395f2b1cfb098b3f1ae89
Jul 28 00:10:08.398996 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.398972    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"]
Jul 28 00:10:08.401159 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:10:08.401130    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac73c56_6096_44d1_9191_7f402995d6b8.slice/crio-c2220ddbffc4cdec0f57fbce51b188da1e918d141655351ac8dd980160789ca9 WatchSource:0}: Error finding container c2220ddbffc4cdec0f57fbce51b188da1e918d141655351ac8dd980160789ca9: Status 404 returned error can't find the container with id c2220ddbffc4cdec0f57fbce51b188da1e918d141655351ac8dd980160789ca9
Jul 28 00:10:08.881153 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.881060    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr" event={"ID":"4ac73c56-6096-44d1-9191-7f402995d6b8","Type":"ContainerStarted","Data":"2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4"}
Jul 28 00:10:08.881153 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.881112    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr" event={"ID":"4ac73c56-6096-44d1-9191-7f402995d6b8","Type":"ContainerStarted","Data":"c2220ddbffc4cdec0f57fbce51b188da1e918d141655351ac8dd980160789ca9"}
Jul 28 00:10:08.881684 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.881240    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:08.882800 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.882778    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf" event={"ID":"0838d5f2-dc6b-4ff8-87b4-86637535be7c","Type":"ContainerStarted","Data":"d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4"}
Jul 28 00:10:08.882911 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.882815    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:08.882911 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.882831    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf" event={"ID":"0838d5f2-dc6b-4ff8-87b4-86637535be7c","Type":"ContainerStarted","Data":"c82951e780ef063080c21911c61807b0b28bc1f5ac1395f2b1cfb098b3f1ae89"}
Jul 28 00:10:08.896674 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.896627    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr" podStartSLOduration=1.8966118779999999 podStartE2EDuration="1.896611878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:10:08.895332855 +0000 UTC m=+3233.846676255" watchObservedRunningTime="2026-07-28 00:10:08.896611878 +0000 UTC m=+3233.847955264"
Jul 28 00:10:08.912389 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:08.912326    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf" podStartSLOduration=1.912310513 podStartE2EDuration="1.912310513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:10:08.910044972 +0000 UTC m=+3233.861388382" watchObservedRunningTime="2026-07-28 00:10:08.912310513 +0000 UTC m=+3233.863653899"
Jul 28 00:10:18.939531 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:18.939496    2570 generic.go:356] "Generic (PLEG): container finished" podID="e161adfa-1131-4054-884e-720b606cca79" containerID="4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231" exitCode=0
Jul 28 00:10:18.939976 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:18.939570    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" event={"ID":"e161adfa-1131-4054-884e-720b606cca79","Type":"ContainerDied","Data":"4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"}
Jul 28 00:10:18.940020 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:18.939974    2570 scope.go:117] "RemoveContainer" containerID="4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"
Jul 28 00:10:18.941102 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:18.941071    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 00:10:18.941208 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:18.941159    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 00:10:18.941270 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:18.941207    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 28 00:10:19.890601 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:19.890568    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:19.890775 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:19.890637    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:19.950287 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:19.950244    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" event={"ID":"e161adfa-1131-4054-884e-720b606cca79","Type":"ContainerStarted","Data":"eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091"}
Jul 28 00:10:20.365476 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:20.365258    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:20.367446 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:20.367411    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 28 00:10:20.367712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:20.367683    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 28 00:10:20.862326 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:20.862264    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:21.364915 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:21.364873    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 28 00:10:21.365518 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:21.365484    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 28 00:10:22.365713 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:22.365675    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 28 00:10:22.366215 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:22.365739    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 28 00:10:23.365509 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:23.365467    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 28 00:10:23.365721 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:23.365534    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 28 00:10:24.369267 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:24.369236    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:24.370229 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:24.370210    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:32.611707 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.611670    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"]
Jul 28 00:10:32.612116 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.611949    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr" podUID="4ac73c56-6096-44d1-9191-7f402995d6b8" containerName="payload-processing" containerID="cri-o://2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4" gracePeriod=30
Jul 28 00:10:32.622516 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.622477    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"]
Jul 28 00:10:32.622847 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.622777    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf" podUID="0838d5f2-dc6b-4ff8-87b4-86637535be7c" containerName="payload-pre-processing" containerID="cri-o://d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4" gracePeriod=30
Jul 28 00:10:32.893016 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.892989    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:32.922139 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.922117    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:32.984940 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.984912    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smfr7\" (UniqueName: \"kubernetes.io/projected/4ac73c56-6096-44d1-9191-7f402995d6b8-kube-api-access-smfr7\") pod \"4ac73c56-6096-44d1-9191-7f402995d6b8\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") "
Jul 28 00:10:32.985108 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.984955    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0838d5f2-dc6b-4ff8-87b4-86637535be7c-plugins-config-volume\") pod \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") "
Jul 28 00:10:32.985108 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.985024    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac73c56-6096-44d1-9191-7f402995d6b8-plugins-config-volume\") pod \"4ac73c56-6096-44d1-9191-7f402995d6b8\" (UID: \"4ac73c56-6096-44d1-9191-7f402995d6b8\") "
Jul 28 00:10:32.985108 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.985047    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs2nj\" (UniqueName: \"kubernetes.io/projected/0838d5f2-dc6b-4ff8-87b4-86637535be7c-kube-api-access-hs2nj\") pod \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\" (UID: \"0838d5f2-dc6b-4ff8-87b4-86637535be7c\") "
Jul 28 00:10:32.985406 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.985384    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0838d5f2-dc6b-4ff8-87b4-86637535be7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0838d5f2-dc6b-4ff8-87b4-86637535be7c" (UID: "0838d5f2-dc6b-4ff8-87b4-86637535be7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:10:32.985472 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.985413    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac73c56-6096-44d1-9191-7f402995d6b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ac73c56-6096-44d1-9191-7f402995d6b8" (UID: "4ac73c56-6096-44d1-9191-7f402995d6b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:10:32.987227 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.987206    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac73c56-6096-44d1-9191-7f402995d6b8-kube-api-access-smfr7" (OuterVolumeSpecName: "kube-api-access-smfr7") pod "4ac73c56-6096-44d1-9191-7f402995d6b8" (UID: "4ac73c56-6096-44d1-9191-7f402995d6b8"). InnerVolumeSpecName "kube-api-access-smfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:10:32.987288 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:32.987270    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0838d5f2-dc6b-4ff8-87b4-86637535be7c-kube-api-access-hs2nj" (OuterVolumeSpecName: "kube-api-access-hs2nj") pod "0838d5f2-dc6b-4ff8-87b4-86637535be7c" (UID: "0838d5f2-dc6b-4ff8-87b4-86637535be7c"). InnerVolumeSpecName "kube-api-access-hs2nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:10:33.035161 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.035127    2570 generic.go:356] "Generic (PLEG): container finished" podID="4ac73c56-6096-44d1-9191-7f402995d6b8" containerID="2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4" exitCode=0
Jul 28 00:10:33.035381 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.035177    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"
Jul 28 00:10:33.035381 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.035223    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr" event={"ID":"4ac73c56-6096-44d1-9191-7f402995d6b8","Type":"ContainerDied","Data":"2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4"}
Jul 28 00:10:33.035381 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.035281    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr" event={"ID":"4ac73c56-6096-44d1-9191-7f402995d6b8","Type":"ContainerDied","Data":"c2220ddbffc4cdec0f57fbce51b188da1e918d141655351ac8dd980160789ca9"}
Jul 28 00:10:33.035381 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.035309    2570 scope.go:117] "RemoveContainer" containerID="2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4"
Jul 28 00:10:33.036653 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.036629    2570 generic.go:356] "Generic (PLEG): container finished" podID="0838d5f2-dc6b-4ff8-87b4-86637535be7c" containerID="d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4" exitCode=0
Jul 28 00:10:33.036743 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.036668    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"
Jul 28 00:10:33.036743 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.036701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf" event={"ID":"0838d5f2-dc6b-4ff8-87b4-86637535be7c","Type":"ContainerDied","Data":"d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4"}
Jul 28 00:10:33.036846 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.036744    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf" event={"ID":"0838d5f2-dc6b-4ff8-87b4-86637535be7c","Type":"ContainerDied","Data":"c82951e780ef063080c21911c61807b0b28bc1f5ac1395f2b1cfb098b3f1ae89"}
Jul 28 00:10:33.046096 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.046079    2570 scope.go:117] "RemoveContainer" containerID="2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4"
Jul 28 00:10:33.046413 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:10:33.046393    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4\": container with ID starting with 2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4 not found: ID does not exist" containerID="2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4"
Jul 28 00:10:33.046504 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.046427    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4"} err="failed to get container status \"2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4\": rpc error: code = NotFound desc = could not find container \"2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4\": container with ID starting with 2634e947da5c3fdc30eb48b459cc06ad83a8e2a61af3b69f78b7c55a451027d4 not found: ID does not exist"
Jul 28 00:10:33.046504 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.046453    2570 scope.go:117] "RemoveContainer" containerID="d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4"
Jul 28 00:10:33.055417 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.055398    2570 scope.go:117] "RemoveContainer" containerID="d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4"
Jul 28 00:10:33.055686 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:10:33.055666    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4\": container with ID starting with d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4 not found: ID does not exist" containerID="d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4"
Jul 28 00:10:33.055752 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.055694    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4"} err="failed to get container status \"d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4\": rpc error: code = NotFound desc = could not find container \"d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4\": container with ID starting with d2754bb28219f000b91bdeaf5eeec91cd7313b195fdcea67dd239e5ce8161fc4 not found: ID does not exist"
Jul 28 00:10:33.060900 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.060879    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"]
Jul 28 00:10:33.064171 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.064149    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35fc9-b8788b97b-wtncf"]
Jul 28 00:10:33.074800 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.074775    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"]
Jul 28 00:10:33.076667 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.076649    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a35fc9-588c674bfd-j58sr"]
Jul 28 00:10:33.086271 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.086245    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac73c56-6096-44d1-9191-7f402995d6b8-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:33.086271 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.086270    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2nj\" (UniqueName: \"kubernetes.io/projected/0838d5f2-dc6b-4ff8-87b4-86637535be7c-kube-api-access-hs2nj\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:33.086407 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.086280    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfr7\" (UniqueName: \"kubernetes.io/projected/4ac73c56-6096-44d1-9191-7f402995d6b8-kube-api-access-smfr7\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:33.086407 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.086290    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0838d5f2-dc6b-4ff8-87b4-86637535be7c-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:33.522285 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.522249    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0838d5f2-dc6b-4ff8-87b4-86637535be7c" path="/var/lib/kubelet/pods/0838d5f2-dc6b-4ff8-87b4-86637535be7c/volumes"
Jul 28 00:10:33.522749 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:33.522734    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac73c56-6096-44d1-9191-7f402995d6b8" path="/var/lib/kubelet/pods/4ac73c56-6096-44d1-9191-7f402995d6b8/volumes"
Jul 28 00:10:43.590944 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:43.590906    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"]
Jul 28 00:10:43.591508 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:43.591167    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy" containerID="cri-o://eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091" gracePeriod=30
Jul 28 00:10:48.882467 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.882442    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:48.950055 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950024    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-envoy\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950214 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950066    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e161adfa-1131-4054-884e-720b606cca79-istiod-ca-cert\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950214 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950139    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-certs\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950214 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950162    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e161adfa-1131-4054-884e-720b606cca79-istio-podinfo\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950214 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950202    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-credential-socket\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950440 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950223    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-socket\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950440 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950245    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptxfn\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-kube-api-access-ptxfn\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950440 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950261    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-data\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950606 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950483    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:10:48.950606 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950499    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:10:48.950606 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950536    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e161adfa-1131-4054-884e-720b606cca79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:10:48.950606 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950578    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-istio-token\") pod \"e161adfa-1131-4054-884e-720b606cca79\" (UID: \"e161adfa-1131-4054-884e-720b606cca79\") "
Jul 28 00:10:48.950853 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.950800    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:10:48.951223 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.951199    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:48.951310 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.951227    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:48.951310 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.951243    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e161adfa-1131-4054-884e-720b606cca79-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:48.951310 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.951258    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:48.951742 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.951714    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-data" (OuterVolumeSpecName: "istio-data") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:10:48.952736 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.952706    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-istio-token" (OuterVolumeSpecName: "istio-token") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:10:48.952844 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.952783    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-kube-api-access-ptxfn" (OuterVolumeSpecName: "kube-api-access-ptxfn") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "kube-api-access-ptxfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:10:48.952901 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.952837    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:10:48.952972 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:48.952945    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e161adfa-1131-4054-884e-720b606cca79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e161adfa-1131-4054-884e-720b606cca79" (UID: "e161adfa-1131-4054-884e-720b606cca79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 00:10:49.052398 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.052339    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e161adfa-1131-4054-884e-720b606cca79-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:49.052398 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.052398    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxfn\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-kube-api-access-ptxfn\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:49.052604 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.052415    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:49.052604 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.052428    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e161adfa-1131-4054-884e-720b606cca79-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:49.052604 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.052440    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e161adfa-1131-4054-884e-720b606cca79-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:10:49.128402 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.128279    2570 generic.go:356] "Generic (PLEG): container finished" podID="e161adfa-1131-4054-884e-720b606cca79" containerID="eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091" exitCode=0
Jul 28 00:10:49.128402 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.128335    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"
Jul 28 00:10:49.128402 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.128338    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" event={"ID":"e161adfa-1131-4054-884e-720b606cca79","Type":"ContainerDied","Data":"eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091"}
Jul 28 00:10:49.128734 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.128450    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl" event={"ID":"e161adfa-1131-4054-884e-720b606cca79","Type":"ContainerDied","Data":"fcc2d76eab896e04aac6eef96ef3fd6f603a53104ab4e766d1c61bba3863e6db"}
Jul 28 00:10:49.128734 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.128485    2570 scope.go:117] "RemoveContainer" containerID="eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091"
Jul 28 00:10:49.138264 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.138244    2570 scope.go:117] "RemoveContainer" containerID="4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"
Jul 28 00:10:49.146884 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.146867    2570 scope.go:117] "RemoveContainer" containerID="eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091"
Jul 28 00:10:49.147140 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:10:49.147121    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091\": container with ID starting with eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091 not found: ID does not exist" containerID="eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091"
Jul 28 00:10:49.147185 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.147150    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091"} err="failed to get container status \"eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091\": rpc error: code = NotFound desc = could not find container \"eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091\": container with ID starting with eed83e643b854981d6c7ff8f7dfa17ea833f68799adee37a41fa98e109a8c091 not found: ID does not exist"
Jul 28 00:10:49.147185 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.147167    2570 scope.go:117] "RemoveContainer" containerID="4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"
Jul 28 00:10:49.147433 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:10:49.147414    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231\": container with ID starting with 4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231 not found: ID does not exist" containerID="4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"
Jul 28 00:10:49.147485 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.147440    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231"} err="failed to get container status \"4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231\": rpc error: code = NotFound desc = could not find container \"4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231\": container with ID starting with 4d2889a926cbe426be14c3edaeb53bc22a17ecfe1bd2abf01fcc6ccebb88f231 not found: ID does not exist"
Jul 28 00:10:49.151248 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.151224    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"]
Jul 28 00:10:49.154790 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.154768    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35fc9-openshift-default-547d4c9db6-69lzl"]
Jul 28 00:10:49.522585 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:10:49.522532    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e161adfa-1131-4054-884e-720b606cca79" path="/var/lib/kubelet/pods/e161adfa-1131-4054-884e-720b606cca79/volumes"
Jul 28 00:11:02.368213 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.368173    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"]
Jul 28 00:11:02.368721 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.368410    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v" podUID="c621b735-7691-4086-9cdc-d6ddd62c7782" containerName="maas-api" containerID="cri-o://fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85" gracePeriod=30
Jul 28 00:11:02.431823 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.431782    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"]
Jul 28 00:11:02.432092 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.432062    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd" podUID="bf035b6c-ae03-4654-b111-bb3db4d51af5" containerName="payload-pre-processing" containerID="cri-o://1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97" gracePeriod=30
Jul 28 00:11:02.697684 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.697653    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:11:02.721328 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.721301    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:11:02.788823 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.788789    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c621b735-7691-4086-9cdc-d6ddd62c7782-maas-api-tls\") pod \"c621b735-7691-4086-9cdc-d6ddd62c7782\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") "
Jul 28 00:11:02.788823 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.788828    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c621b735-7691-4086-9cdc-d6ddd62c7782-openshift-service-ca\") pod \"c621b735-7691-4086-9cdc-d6ddd62c7782\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") "
Jul 28 00:11:02.789085 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.788895    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/c621b735-7691-4086-9cdc-d6ddd62c7782-kube-api-access-bgqrz\") pod \"c621b735-7691-4086-9cdc-d6ddd62c7782\" (UID: \"c621b735-7691-4086-9cdc-d6ddd62c7782\") "
Jul 28 00:11:02.789285 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.789259    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c621b735-7691-4086-9cdc-d6ddd62c7782-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c621b735-7691-4086-9cdc-d6ddd62c7782" (UID: "c621b735-7691-4086-9cdc-d6ddd62c7782"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:11:02.790948 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.790926    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c621b735-7691-4086-9cdc-d6ddd62c7782-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c621b735-7691-4086-9cdc-d6ddd62c7782" (UID: "c621b735-7691-4086-9cdc-d6ddd62c7782"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:11:02.791038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.791018    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c621b735-7691-4086-9cdc-d6ddd62c7782-kube-api-access-bgqrz" (OuterVolumeSpecName: "kube-api-access-bgqrz") pod "c621b735-7691-4086-9cdc-d6ddd62c7782" (UID: "c621b735-7691-4086-9cdc-d6ddd62c7782"). InnerVolumeSpecName "kube-api-access-bgqrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:11:02.889757 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.889728    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf035b6c-ae03-4654-b111-bb3db4d51af5-plugins-config-volume\") pod \"bf035b6c-ae03-4654-b111-bb3db4d51af5\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") "
Jul 28 00:11:02.889880 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.889764    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl6fr\" (UniqueName: \"kubernetes.io/projected/bf035b6c-ae03-4654-b111-bb3db4d51af5-kube-api-access-jl6fr\") pod \"bf035b6c-ae03-4654-b111-bb3db4d51af5\" (UID: \"bf035b6c-ae03-4654-b111-bb3db4d51af5\") "
Jul 28 00:11:02.890058 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.890039    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf035b6c-ae03-4654-b111-bb3db4d51af5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf035b6c-ae03-4654-b111-bb3db4d51af5" (UID: "bf035b6c-ae03-4654-b111-bb3db4d51af5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:11:02.890102 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.890054    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/c621b735-7691-4086-9cdc-d6ddd62c7782-kube-api-access-bgqrz\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:11:02.890102 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.890073    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c621b735-7691-4086-9cdc-d6ddd62c7782-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:11:02.890102 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.890090    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c621b735-7691-4086-9cdc-d6ddd62c7782-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:11:02.891977 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.891960    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf035b6c-ae03-4654-b111-bb3db4d51af5-kube-api-access-jl6fr" (OuterVolumeSpecName: "kube-api-access-jl6fr") pod "bf035b6c-ae03-4654-b111-bb3db4d51af5" (UID: "bf035b6c-ae03-4654-b111-bb3db4d51af5"). InnerVolumeSpecName "kube-api-access-jl6fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:11:02.991053 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.990962    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf035b6c-ae03-4654-b111-bb3db4d51af5-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:11:02.991053 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:02.990995    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl6fr\" (UniqueName: \"kubernetes.io/projected/bf035b6c-ae03-4654-b111-bb3db4d51af5-kube-api-access-jl6fr\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:11:03.199478 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.199442    2570 generic.go:356] "Generic (PLEG): container finished" podID="c621b735-7691-4086-9cdc-d6ddd62c7782" containerID="fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85" exitCode=0
Jul 28 00:11:03.199674 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.199499    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"
Jul 28 00:11:03.199674 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.199522    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v" event={"ID":"c621b735-7691-4086-9cdc-d6ddd62c7782","Type":"ContainerDied","Data":"fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85"}
Jul 28 00:11:03.199674 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.199572    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v" event={"ID":"c621b735-7691-4086-9cdc-d6ddd62c7782","Type":"ContainerDied","Data":"7e65ff5e9fe3473078961b566752e6bee7739d1cdb6b2117c4db70dc3b5c04dd"}
Jul 28 00:11:03.199674 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.199602    2570 scope.go:117] "RemoveContainer" containerID="fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85"
Jul 28 00:11:03.200869 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.200850    2570 generic.go:356] "Generic (PLEG): container finished" podID="bf035b6c-ae03-4654-b111-bb3db4d51af5" containerID="1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97" exitCode=0
Jul 28 00:11:03.200959 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.200908    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"
Jul 28 00:11:03.200959 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.200932    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd" event={"ID":"bf035b6c-ae03-4654-b111-bb3db4d51af5","Type":"ContainerDied","Data":"1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97"}
Jul 28 00:11:03.201079 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.200970    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd" event={"ID":"bf035b6c-ae03-4654-b111-bb3db4d51af5","Type":"ContainerDied","Data":"0824c2a66e18c3b016c3bd4b66cf60efe3feb2747f182280b6bfeb616552148e"}
Jul 28 00:11:03.209443 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.209425    2570 scope.go:117] "RemoveContainer" containerID="fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85"
Jul 28 00:11:03.209731 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:11:03.209711    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85\": container with ID starting with fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85 not found: ID does not exist" containerID="fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85"
Jul 28 00:11:03.209800 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.209740    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85"} err="failed to get container status \"fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85\": rpc error: code = NotFound desc = could not find container \"fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85\": container with ID starting with fffbe04ec3b46d5682a3f85d69e3aa1d0792d16efad9a70e9137022828474a85 not found: ID does not exist"
Jul 28 00:11:03.209800 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.209756    2570 scope.go:117] "RemoveContainer" containerID="1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97"
Jul 28 00:11:03.218152 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.218131    2570 scope.go:117] "RemoveContainer" containerID="1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97"
Jul 28 00:11:03.218447 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:11:03.218425    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97\": container with ID starting with 1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97 not found: ID does not exist" containerID="1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97"
Jul 28 00:11:03.218500 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.218460    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97"} err="failed to get container status \"1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97\": rpc error: code = NotFound desc = could not find container \"1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97\": container with ID starting with 1b217bf0d9d431e2ff6b4754acba0567c4efb1f9374b349986db591036770d97 not found: ID does not exist"
Jul 28 00:11:03.224728 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.224707    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"]
Jul 28 00:11:03.228145 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.228124    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4a6d94-8564994bbc-d7c4v"]
Jul 28 00:11:03.238681 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.238644    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"]
Jul 28 00:11:03.242676 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.242614    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4a6d94-6f96868cb8-8fnkd"]
Jul 28 00:11:03.522396 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.522297    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf035b6c-ae03-4654-b111-bb3db4d51af5" path="/var/lib/kubelet/pods/bf035b6c-ae03-4654-b111-bb3db4d51af5/volumes"
Jul 28 00:11:03.522874 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:03.522856    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c621b735-7691-4086-9cdc-d6ddd62c7782" path="/var/lib/kubelet/pods/c621b735-7691-4086-9cdc-d6ddd62c7782/volumes"
Jul 28 00:11:15.684570 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:15.684542    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:11:15.687658 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:11:15.687637    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:12:17.312275 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.312229    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"]
Jul 28 00:12:17.312929 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.312906    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy"
Jul 28 00:12:17.312981 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.312934    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy"
Jul 28 00:12:17.312981 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.312955    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0838d5f2-dc6b-4ff8-87b4-86637535be7c" containerName="payload-pre-processing"
Jul 28 00:12:17.312981 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.312966    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0838d5f2-dc6b-4ff8-87b4-86637535be7c" containerName="payload-pre-processing"
Jul 28 00:12:17.313075 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.312994    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf035b6c-ae03-4654-b111-bb3db4d51af5" containerName="payload-pre-processing"
Jul 28 00:12:17.313075 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313004    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf035b6c-ae03-4654-b111-bb3db4d51af5" containerName="payload-pre-processing"
Jul 28 00:12:17.313075 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313023    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c621b735-7691-4086-9cdc-d6ddd62c7782" containerName="maas-api"
Jul 28 00:12:17.313075 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313032    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c621b735-7691-4086-9cdc-d6ddd62c7782" containerName="maas-api"
Jul 28 00:12:17.313075 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313052    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac73c56-6096-44d1-9191-7f402995d6b8" containerName="payload-processing"
Jul 28 00:12:17.313075 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313062    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac73c56-6096-44d1-9191-7f402995d6b8" containerName="payload-processing"
Jul 28 00:12:17.313262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313163    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c621b735-7691-4086-9cdc-d6ddd62c7782" containerName="maas-api"
Jul 28 00:12:17.313262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313179    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy"
Jul 28 00:12:17.313262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313191    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy"
Jul 28 00:12:17.313262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313202    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf035b6c-ae03-4654-b111-bb3db4d51af5" containerName="payload-pre-processing"
Jul 28 00:12:17.313262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313212    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0838d5f2-dc6b-4ff8-87b4-86637535be7c" containerName="payload-pre-processing"
Jul 28 00:12:17.313262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.313222    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac73c56-6096-44d1-9191-7f402995d6b8" containerName="payload-processing"
Jul 28 00:12:17.316839 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.316819    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:17.319649 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.319610    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\""
Jul 28 00:12:17.321776 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.321748    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"]
Jul 28 00:12:17.368729 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.368693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de7a8e6-6b76-4c00-be63-be7411b6567f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv\" (UID: \"2de7a8e6-6b76-4c00-be63-be7411b6567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:17.470111 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.470077    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de7a8e6-6b76-4c00-be63-be7411b6567f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv\" (UID: \"2de7a8e6-6b76-4c00-be63-be7411b6567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:17.470743 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.470724    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de7a8e6-6b76-4c00-be63-be7411b6567f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv\" (UID: \"2de7a8e6-6b76-4c00-be63-be7411b6567f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:17.631537 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.631458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:17.767953 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:17.767926    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"]
Jul 28 00:12:17.770970 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:12:17.770937    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de7a8e6_6b76_4c00_be63_be7411b6567f.slice/crio-9f79cfc7cfc44442f1ad6ac2948d2dc4c7ce9abcc50478ed2dd3cafe36eef7cd WatchSource:0}: Error finding container 9f79cfc7cfc44442f1ad6ac2948d2dc4c7ce9abcc50478ed2dd3cafe36eef7cd: Status 404 returned error can't find the container with id 9f79cfc7cfc44442f1ad6ac2948d2dc4c7ce9abcc50478ed2dd3cafe36eef7cd
Jul 28 00:12:18.591491 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:18.591456    2570 generic.go:356] "Generic (PLEG): container finished" podID="2de7a8e6-6b76-4c00-be63-be7411b6567f" containerID="be412a58814c7880971a3081a894124801453d76220304fccfa2f55ade2cc984" exitCode=0
Jul 28 00:12:18.591890 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:18.591545    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv" event={"ID":"2de7a8e6-6b76-4c00-be63-be7411b6567f","Type":"ContainerDied","Data":"be412a58814c7880971a3081a894124801453d76220304fccfa2f55ade2cc984"}
Jul 28 00:12:18.591890 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:18.591617    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv" event={"ID":"2de7a8e6-6b76-4c00-be63-be7411b6567f","Type":"ContainerStarted","Data":"9f79cfc7cfc44442f1ad6ac2948d2dc4c7ce9abcc50478ed2dd3cafe36eef7cd"}
Jul 28 00:12:19.737480 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:19.737458    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:19.792753 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:19.792722    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de7a8e6-6b76-4c00-be63-be7411b6567f-maas-api-service-ca\") pod \"2de7a8e6-6b76-4c00-be63-be7411b6567f\" (UID: \"2de7a8e6-6b76-4c00-be63-be7411b6567f\") "
Jul 28 00:12:19.793103 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:19.793084    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de7a8e6-6b76-4c00-be63-be7411b6567f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2de7a8e6-6b76-4c00-be63-be7411b6567f" (UID: "2de7a8e6-6b76-4c00-be63-be7411b6567f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:19.894259 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:19.894231    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de7a8e6-6b76-4c00-be63-be7411b6567f-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:20.602949 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:20.602912    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"
Jul 28 00:12:20.603140 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:20.602938    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv" event={"ID":"2de7a8e6-6b76-4c00-be63-be7411b6567f","Type":"ContainerDied","Data":"9f79cfc7cfc44442f1ad6ac2948d2dc4c7ce9abcc50478ed2dd3cafe36eef7cd"}
Jul 28 00:12:20.603140 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:20.603010    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f79cfc7cfc44442f1ad6ac2948d2dc4c7ce9abcc50478ed2dd3cafe36eef7cd"
Jul 28 00:12:27.779617 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:27.779578    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"]
Jul 28 00:12:27.780077 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:27.779789    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz" podUID="a6abc96b-6ad3-4d5b-9add-38349c43bd26" containerName="maas-api" containerID="cri-o://08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70" gracePeriod=30
Jul 28 00:12:27.827345 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:27.827313    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"]
Jul 28 00:12:27.827597 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:27.827572    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm" podUID="c481186c-38f5-4aa7-a502-bc4f0a652044" containerName="payload-processing" containerID="cri-o://73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc" gracePeriod=30
Jul 28 00:12:27.833655 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:27.832958    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"]
Jul 28 00:12:27.833655 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:27.833177    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr" podUID="00e2931b-e84b-46bd-91f2-f52ff4697b31" containerName="payload-pre-processing" containerID="cri-o://a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0" gracePeriod=30
Jul 28 00:12:28.125002 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.124977    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 28 00:12:28.172296 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.172269    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 28 00:12:28.175595 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.175573    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 28 00:12:28.296326 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296290    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e2931b-e84b-46bd-91f2-f52ff4697b31-plugins-config-volume\") pod \"00e2931b-e84b-46bd-91f2-f52ff4697b31\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") "
Jul 28 00:12:28.296588 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296428    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kghgr\" (UniqueName: \"kubernetes.io/projected/a6abc96b-6ad3-4d5b-9add-38349c43bd26-kube-api-access-kghgr\") pod \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") "
Jul 28 00:12:28.296588 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296502    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6abc96b-6ad3-4d5b-9add-38349c43bd26-maas-api-tls\") pod \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") "
Jul 28 00:12:28.296588 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296532    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7j5h\" (UniqueName: \"kubernetes.io/projected/c481186c-38f5-4aa7-a502-bc4f0a652044-kube-api-access-g7j5h\") pod \"c481186c-38f5-4aa7-a502-bc4f0a652044\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") "
Jul 28 00:12:28.296760 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296588    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5psl5\" (UniqueName: \"kubernetes.io/projected/00e2931b-e84b-46bd-91f2-f52ff4697b31-kube-api-access-5psl5\") pod \"00e2931b-e84b-46bd-91f2-f52ff4697b31\" (UID: \"00e2931b-e84b-46bd-91f2-f52ff4697b31\") "
Jul 28 00:12:28.296760 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296625    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6abc96b-6ad3-4d5b-9add-38349c43bd26-openshift-service-ca\") pod \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\" (UID: \"a6abc96b-6ad3-4d5b-9add-38349c43bd26\") "
Jul 28 00:12:28.296760 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296655    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c481186c-38f5-4aa7-a502-bc4f0a652044-plugins-config-volume\") pod \"c481186c-38f5-4aa7-a502-bc4f0a652044\" (UID: \"c481186c-38f5-4aa7-a502-bc4f0a652044\") "
Jul 28 00:12:28.296986 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296956    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e2931b-e84b-46bd-91f2-f52ff4697b31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00e2931b-e84b-46bd-91f2-f52ff4697b31" (UID: "00e2931b-e84b-46bd-91f2-f52ff4697b31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:28.297090 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.296992    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6abc96b-6ad3-4d5b-9add-38349c43bd26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6abc96b-6ad3-4d5b-9add-38349c43bd26" (UID: "a6abc96b-6ad3-4d5b-9add-38349c43bd26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:28.297302 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.297268    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c481186c-38f5-4aa7-a502-bc4f0a652044-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c481186c-38f5-4aa7-a502-bc4f0a652044" (UID: "c481186c-38f5-4aa7-a502-bc4f0a652044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:28.298927 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.298892    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e2931b-e84b-46bd-91f2-f52ff4697b31-kube-api-access-5psl5" (OuterVolumeSpecName: "kube-api-access-5psl5") pod "00e2931b-e84b-46bd-91f2-f52ff4697b31" (UID: "00e2931b-e84b-46bd-91f2-f52ff4697b31"). InnerVolumeSpecName "kube-api-access-5psl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:28.299204 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.299184    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6abc96b-6ad3-4d5b-9add-38349c43bd26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6abc96b-6ad3-4d5b-9add-38349c43bd26" (UID: "a6abc96b-6ad3-4d5b-9add-38349c43bd26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:12:28.299331 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.299311    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c481186c-38f5-4aa7-a502-bc4f0a652044-kube-api-access-g7j5h" (OuterVolumeSpecName: "kube-api-access-g7j5h") pod "c481186c-38f5-4aa7-a502-bc4f0a652044" (UID: "c481186c-38f5-4aa7-a502-bc4f0a652044"). InnerVolumeSpecName "kube-api-access-g7j5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:28.299400 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.299323    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6abc96b-6ad3-4d5b-9add-38349c43bd26-kube-api-access-kghgr" (OuterVolumeSpecName: "kube-api-access-kghgr") pod "a6abc96b-6ad3-4d5b-9add-38349c43bd26" (UID: "a6abc96b-6ad3-4d5b-9add-38349c43bd26"). InnerVolumeSpecName "kube-api-access-kghgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:28.398262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398229    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e2931b-e84b-46bd-91f2-f52ff4697b31-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.398262 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398260    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghgr\" (UniqueName: \"kubernetes.io/projected/a6abc96b-6ad3-4d5b-9add-38349c43bd26-kube-api-access-kghgr\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.398520 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398270    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6abc96b-6ad3-4d5b-9add-38349c43bd26-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.398520 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398280    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7j5h\" (UniqueName: \"kubernetes.io/projected/c481186c-38f5-4aa7-a502-bc4f0a652044-kube-api-access-g7j5h\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.398520 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398291    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psl5\" (UniqueName: \"kubernetes.io/projected/00e2931b-e84b-46bd-91f2-f52ff4697b31-kube-api-access-5psl5\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.398520 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398299    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6abc96b-6ad3-4d5b-9add-38349c43bd26-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.398520 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.398309    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c481186c-38f5-4aa7-a502-bc4f0a652044-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:28.644031 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.644000    2570 generic.go:356] "Generic (PLEG): container finished" podID="a6abc96b-6ad3-4d5b-9add-38349c43bd26" containerID="08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70" exitCode=0
Jul 28 00:12:28.644245 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.644060    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"
Jul 28 00:12:28.644245 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.644098    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz" event={"ID":"a6abc96b-6ad3-4d5b-9add-38349c43bd26","Type":"ContainerDied","Data":"08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70"}
Jul 28 00:12:28.644245 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.644151    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz" event={"ID":"a6abc96b-6ad3-4d5b-9add-38349c43bd26","Type":"ContainerDied","Data":"3e98e6f000b1e5ee7c261fefc5580710f7d43e57def4b750a551b7c5637bfae6"}
Jul 28 00:12:28.644245 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.644182    2570 scope.go:117] "RemoveContainer" containerID="08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70"
Jul 28 00:12:28.645497 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.645476    2570 generic.go:356] "Generic (PLEG): container finished" podID="c481186c-38f5-4aa7-a502-bc4f0a652044" containerID="73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc" exitCode=0
Jul 28 00:12:28.645576 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.645507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm" event={"ID":"c481186c-38f5-4aa7-a502-bc4f0a652044","Type":"ContainerDied","Data":"73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc"}
Jul 28 00:12:28.645640 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.645578    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm" event={"ID":"c481186c-38f5-4aa7-a502-bc4f0a652044","Type":"ContainerDied","Data":"5309e4610de1cec5c69f0de7d7b9b8b454195b18d166f8bea19accc8420786b9"}
Jul 28 00:12:28.645640 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.645520    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"
Jul 28 00:12:28.646934 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.646911    2570 generic.go:356] "Generic (PLEG): container finished" podID="00e2931b-e84b-46bd-91f2-f52ff4697b31" containerID="a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0" exitCode=0
Jul 28 00:12:28.647043 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.646949    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"
Jul 28 00:12:28.647043 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.647022    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr" event={"ID":"00e2931b-e84b-46bd-91f2-f52ff4697b31","Type":"ContainerDied","Data":"a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0"}
Jul 28 00:12:28.647165 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.647090    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr" event={"ID":"00e2931b-e84b-46bd-91f2-f52ff4697b31","Type":"ContainerDied","Data":"f0f97996fad111e8b42872061f89733bb5cedf8d618e7517b9ea13dd51c51faa"}
Jul 28 00:12:28.658451 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.658432    2570 scope.go:117] "RemoveContainer" containerID="08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70"
Jul 28 00:12:28.658747 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:12:28.658728    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70\": container with ID starting with 08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70 not found: ID does not exist" containerID="08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70"
Jul 28 00:12:28.658809 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.658755    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70"} err="failed to get container status \"08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70\": rpc error: code = NotFound desc = could not find container \"08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70\": container with ID starting with 08d12cd9353dbed6d29ef27025f04467649d0f7cfaae8a504aea5f8b2dd11e70 not found: ID does not exist"
Jul 28 00:12:28.658809 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.658773    2570 scope.go:117] "RemoveContainer" containerID="73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc"
Jul 28 00:12:28.667117 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.667099    2570 scope.go:117] "RemoveContainer" containerID="73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc"
Jul 28 00:12:28.667442 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:12:28.667419    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc\": container with ID starting with 73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc not found: ID does not exist" containerID="73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc"
Jul 28 00:12:28.667499 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.667450    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc"} err="failed to get container status \"73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc\": rpc error: code = NotFound desc = could not find container \"73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc\": container with ID starting with 73a05d56d237710c806bb2a3b04b6731061111400c920bc410f82bdce2960bcc not found: ID does not exist"
Jul 28 00:12:28.667499 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.667471    2570 scope.go:117] "RemoveContainer" containerID="a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0"
Jul 28 00:12:28.670953 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.670934    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"]
Jul 28 00:12:28.673996 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.673974    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-42d790-765d65bc68-jzmnr"]
Jul 28 00:12:28.676575 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.676558    2570 scope.go:117] "RemoveContainer" containerID="a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0"
Jul 28 00:12:28.676847 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:12:28.676828    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0\": container with ID starting with a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0 not found: ID does not exist" containerID="a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0"
Jul 28 00:12:28.676895 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.676855    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0"} err="failed to get container status \"a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0\": rpc error: code = NotFound desc = could not find container \"a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0\": container with ID starting with a65c6e0f60a790c9aa0fb3a513d46669eb1fe93f51450195538e79f974f62ab0 not found: ID does not exist"
Jul 28 00:12:28.684069 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.684048    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"]
Jul 28 00:12:28.688752 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.688734    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-42d790-78df554c4f-n67gz"]
Jul 28 00:12:28.698930 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.698908    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"]
Jul 28 00:12:28.704948 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:28.704925    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-42d790-588bccdc98-zlhzm"]
Jul 28 00:12:29.522246 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:29.522214    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e2931b-e84b-46bd-91f2-f52ff4697b31" path="/var/lib/kubelet/pods/00e2931b-e84b-46bd-91f2-f52ff4697b31/volumes"
Jul 28 00:12:29.522678 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:29.522662    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6abc96b-6ad3-4d5b-9add-38349c43bd26" path="/var/lib/kubelet/pods/a6abc96b-6ad3-4d5b-9add-38349c43bd26/volumes"
Jul 28 00:12:29.522994 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:29.522974    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c481186c-38f5-4aa7-a502-bc4f0a652044" path="/var/lib/kubelet/pods/c481186c-38f5-4aa7-a502-bc4f0a652044/volumes"
Jul 28 00:12:32.372679 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:32.372637    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"]
Jul 28 00:12:32.374641 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:32.374611    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-42d790-7hvzv"]
Jul 28 00:12:33.522448 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:33.522417    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de7a8e6-6b76-4c00-be63-be7411b6567f" path="/var/lib/kubelet/pods/2de7a8e6-6b76-4c00-be63-be7411b6567f/volumes"
Jul 28 00:12:38.787717 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:38.787680    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"]
Jul 28 00:12:38.788126 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:38.787936    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy" containerID="cri-o://f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554" gracePeriod=30
Jul 28 00:12:44.036036 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.036005    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 28 00:12:44.138841 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.138808    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-data\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.138854    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-certs\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.138888    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-envoy\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.138912    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-credential-socket\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.138937    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-token\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.138968    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llvhb\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-kube-api-access-llvhb\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139301 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139122    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-podinfo\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139301 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139143    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:44.139301 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139231    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istiod-ca-cert\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139301 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139239    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:44.139301 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139286    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-socket\") pod \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\" (UID: \"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05\") "
Jul 28 00:12:44.139598 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139562    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:44.139657 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139647    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istiod-ca-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.139710 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139661    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.139710 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139670    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-credential-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.139710 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139713    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:44.140025 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.139747    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:44.141406 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.141380    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:12:44.141522 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.141507    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 00:12:44.141791 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.141767    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:44.141883 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.141797    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-kube-api-access-llvhb" (OuterVolumeSpecName: "kube-api-access-llvhb") pod "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" (UID: "b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05"). InnerVolumeSpecName "kube-api-access-llvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:44.240427 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.240382    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-podinfo\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.240427 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.240425    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-workload-socket\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.240637 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.240444    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-data\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.240637 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.240455    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-envoy\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.240637 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.240463    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.240637 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.240474    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llvhb\" (UniqueName: \"kubernetes.io/projected/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05-kube-api-access-llvhb\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:44.733026 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.732992    2570 generic.go:356] "Generic (PLEG): container finished" podID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerID="f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554" exitCode=0
Jul 28 00:12:44.733196 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.733060    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"
Jul 28 00:12:44.733196 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.733084    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" event={"ID":"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05","Type":"ContainerDied","Data":"f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554"}
Jul 28 00:12:44.733196 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.733175    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx" event={"ID":"b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05","Type":"ContainerDied","Data":"98f13ff5f8c4692c67bb0e0241244873bcfb99a6465056107dd39a229a28c1e4"}
Jul 28 00:12:44.733302 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.733204    2570 scope.go:117] "RemoveContainer" containerID="f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554"
Jul 28 00:12:44.742758 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.742739    2570 scope.go:117] "RemoveContainer" containerID="0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"
Jul 28 00:12:44.751011 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.750982    2570 scope.go:117] "RemoveContainer" containerID="f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554"
Jul 28 00:12:44.751267 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:12:44.751244    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554\": container with ID starting with f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554 not found: ID does not exist" containerID="f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554"
Jul 28 00:12:44.751332 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.751276    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554"} err="failed to get container status \"f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554\": rpc error: code = NotFound desc = could not find container \"f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554\": container with ID starting with f8360dbcdab1cd178fe9bd02002c241fb2553444e9e414be9f88c473255a9554 not found: ID does not exist"
Jul 28 00:12:44.751332 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.751295    2570 scope.go:117] "RemoveContainer" containerID="0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"
Jul 28 00:12:44.751552 ip-10-0-136-102 kubenswrapper[2570]: E0728 00:12:44.751524    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd\": container with ID starting with 0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd not found: ID does not exist" containerID="0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"
Jul 28 00:12:44.751676 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.751553    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd"} err="failed to get container status \"0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd\": rpc error: code = NotFound desc = could not find container \"0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd\": container with ID starting with 0ab5b977a089a516e52ed4677a2f2746982390edbdfc545970696788b3da9fdd not found: ID does not exist"
Jul 28 00:12:44.754393 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.754373    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"]
Jul 28 00:12:44.757866 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:44.757847    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-42d790-openshift-default-5f9d44f488-p9zpx"]
Jul 28 00:12:45.521844 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:45.521809    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" path="/var/lib/kubelet/pods/b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05/volumes"
Jul 28 00:12:49.489712 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.489682    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"]
Jul 28 00:12:49.490313 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.489938    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q" podUID="bf2c0064-2eff-4db3-944a-8f11f84ae59c" containerName="maas-api" containerID="cri-o://8f1fdd2941e20f9d66652fcd1c5a3f5932235fb43074bcbda9fa634cbffa569e" gracePeriod=30
Jul 28 00:12:49.543234 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.543200    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"]
Jul 28 00:12:49.543488 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.543462    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6" podUID="4b12240f-44aa-48a2-bad9-8f7fed96987b" containerName="payload-processing" containerID="cri-o://7939e2cddccb0264d5538c4e2be5445f5b3ef388d8514d13a4ede87f4690e96d" gracePeriod=30
Jul 28 00:12:49.551168 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.551142    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"]
Jul 28 00:12:49.551454 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.551416    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt" podUID="e523ad06-3470-421f-b533-0b023e108cb4" containerName="payload-pre-processing" containerID="cri-o://94d9c31d8991bae2c399af7363eaee25da333a6ac015d2aa3b33feee13dd77f1" gracePeriod=30
Jul 28 00:12:49.761475 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.761367    2570 generic.go:356] "Generic (PLEG): container finished" podID="4b12240f-44aa-48a2-bad9-8f7fed96987b" containerID="7939e2cddccb0264d5538c4e2be5445f5b3ef388d8514d13a4ede87f4690e96d" exitCode=0
Jul 28 00:12:49.761475 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.761396    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6" event={"ID":"4b12240f-44aa-48a2-bad9-8f7fed96987b","Type":"ContainerDied","Data":"7939e2cddccb0264d5538c4e2be5445f5b3ef388d8514d13a4ede87f4690e96d"}
Jul 28 00:12:49.763179 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.763157    2570 generic.go:356] "Generic (PLEG): container finished" podID="e523ad06-3470-421f-b533-0b023e108cb4" containerID="94d9c31d8991bae2c399af7363eaee25da333a6ac015d2aa3b33feee13dd77f1" exitCode=0
Jul 28 00:12:49.763304 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.763198    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt" event={"ID":"e523ad06-3470-421f-b533-0b023e108cb4","Type":"ContainerDied","Data":"94d9c31d8991bae2c399af7363eaee25da333a6ac015d2aa3b33feee13dd77f1"}
Jul 28 00:12:49.765029 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.765003    2570 generic.go:356] "Generic (PLEG): container finished" podID="bf2c0064-2eff-4db3-944a-8f11f84ae59c" containerID="8f1fdd2941e20f9d66652fcd1c5a3f5932235fb43074bcbda9fa634cbffa569e" exitCode=0
Jul 28 00:12:49.765127 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.765058    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q" event={"ID":"bf2c0064-2eff-4db3-944a-8f11f84ae59c","Type":"ContainerDied","Data":"8f1fdd2941e20f9d66652fcd1c5a3f5932235fb43074bcbda9fa634cbffa569e"}
Jul 28 00:12:49.803285 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.803255    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 28 00:12:49.822990 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.821682    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 28 00:12:49.851333 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.851301    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 28 00:12:49.890720 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.890688    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2c0064-2eff-4db3-944a-8f11f84ae59c-openshift-service-ca\") pod \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") "
Jul 28 00:12:49.890903 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.890758    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74sjl\" (UniqueName: \"kubernetes.io/projected/4b12240f-44aa-48a2-bad9-8f7fed96987b-kube-api-access-74sjl\") pod \"4b12240f-44aa-48a2-bad9-8f7fed96987b\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") "
Jul 28 00:12:49.890903 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.890806    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2c0064-2eff-4db3-944a-8f11f84ae59c-maas-api-tls\") pod \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") "
Jul 28 00:12:49.891037 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.890900    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b12240f-44aa-48a2-bad9-8f7fed96987b-plugins-config-volume\") pod \"4b12240f-44aa-48a2-bad9-8f7fed96987b\" (UID: \"4b12240f-44aa-48a2-bad9-8f7fed96987b\") "
Jul 28 00:12:49.891037 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.890940    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6ksw\" (UniqueName: \"kubernetes.io/projected/bf2c0064-2eff-4db3-944a-8f11f84ae59c-kube-api-access-d6ksw\") pod \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\" (UID: \"bf2c0064-2eff-4db3-944a-8f11f84ae59c\") "
Jul 28 00:12:49.891225 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.891114    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2c0064-2eff-4db3-944a-8f11f84ae59c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf2c0064-2eff-4db3-944a-8f11f84ae59c" (UID: "bf2c0064-2eff-4db3-944a-8f11f84ae59c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:49.891337 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.891316    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b12240f-44aa-48a2-bad9-8f7fed96987b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b12240f-44aa-48a2-bad9-8f7fed96987b" (UID: "4b12240f-44aa-48a2-bad9-8f7fed96987b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:49.891428 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.891340    2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2c0064-2eff-4db3-944a-8f11f84ae59c-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:49.892990 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.892970    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b12240f-44aa-48a2-bad9-8f7fed96987b-kube-api-access-74sjl" (OuterVolumeSpecName: "kube-api-access-74sjl") pod "4b12240f-44aa-48a2-bad9-8f7fed96987b" (UID: "4b12240f-44aa-48a2-bad9-8f7fed96987b"). InnerVolumeSpecName "kube-api-access-74sjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:49.893126 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.893101    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2c0064-2eff-4db3-944a-8f11f84ae59c-kube-api-access-d6ksw" (OuterVolumeSpecName: "kube-api-access-d6ksw") pod "bf2c0064-2eff-4db3-944a-8f11f84ae59c" (UID: "bf2c0064-2eff-4db3-944a-8f11f84ae59c"). InnerVolumeSpecName "kube-api-access-d6ksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:49.893274 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.893251    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2c0064-2eff-4db3-944a-8f11f84ae59c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf2c0064-2eff-4db3-944a-8f11f84ae59c" (UID: "bf2c0064-2eff-4db3-944a-8f11f84ae59c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:12:49.992029 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.991996    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e523ad06-3470-421f-b533-0b023e108cb4-plugins-config-volume\") pod \"e523ad06-3470-421f-b533-0b023e108cb4\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") "
Jul 28 00:12:49.992228 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.992054    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts2nt\" (UniqueName: \"kubernetes.io/projected/e523ad06-3470-421f-b533-0b023e108cb4-kube-api-access-ts2nt\") pod \"e523ad06-3470-421f-b533-0b023e108cb4\" (UID: \"e523ad06-3470-421f-b533-0b023e108cb4\") "
Jul 28 00:12:49.992228 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.992219    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74sjl\" (UniqueName: \"kubernetes.io/projected/4b12240f-44aa-48a2-bad9-8f7fed96987b-kube-api-access-74sjl\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:49.992307 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.992230    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf2c0064-2eff-4db3-944a-8f11f84ae59c-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:49.992307 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.992238    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b12240f-44aa-48a2-bad9-8f7fed96987b-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:49.992307 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.992247    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ksw\" (UniqueName: \"kubernetes.io/projected/bf2c0064-2eff-4db3-944a-8f11f84ae59c-kube-api-access-d6ksw\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:49.992487 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.992428    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e523ad06-3470-421f-b533-0b023e108cb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e523ad06-3470-421f-b533-0b023e108cb4" (UID: "e523ad06-3470-421f-b533-0b023e108cb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:12:49.994246 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:49.994228    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e523ad06-3470-421f-b533-0b023e108cb4-kube-api-access-ts2nt" (OuterVolumeSpecName: "kube-api-access-ts2nt") pod "e523ad06-3470-421f-b533-0b023e108cb4" (UID: "e523ad06-3470-421f-b533-0b023e108cb4"). InnerVolumeSpecName "kube-api-access-ts2nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:12:50.092969 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.092895    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts2nt\" (UniqueName: \"kubernetes.io/projected/e523ad06-3470-421f-b533-0b023e108cb4-kube-api-access-ts2nt\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:50.092969 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.092919    2570 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e523ad06-3470-421f-b533-0b023e108cb4-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\""
Jul 28 00:12:50.777026 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.776942    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"
Jul 28 00:12:50.777478 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.776939    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6" event={"ID":"4b12240f-44aa-48a2-bad9-8f7fed96987b","Type":"ContainerDied","Data":"58ee9ed3cee3b5365c6d83a74f14af4051a2735bff490f0b16f460f45f3c5e7b"}
Jul 28 00:12:50.777478 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.777122    2570 scope.go:117] "RemoveContainer" containerID="7939e2cddccb0264d5538c4e2be5445f5b3ef388d8514d13a4ede87f4690e96d"
Jul 28 00:12:50.778478 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.778392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt" event={"ID":"e523ad06-3470-421f-b533-0b023e108cb4","Type":"ContainerDied","Data":"e6e9f9c34a6cc56de10a4bdaaaab9d308c833cf34299d4083e2494d25cab93e9"}
Jul 28 00:12:50.778478 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.778412    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"
Jul 28 00:12:50.779959 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.779918    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q" event={"ID":"bf2c0064-2eff-4db3-944a-8f11f84ae59c","Type":"ContainerDied","Data":"de6bdaaa5cbff3f10e04d534f2c452955036d2858a7540141e624d3ae4539df8"}
Jul 28 00:12:50.780085 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.779962    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"
Jul 28 00:12:50.787576 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.787554    2570 scope.go:117] "RemoveContainer" containerID="94d9c31d8991bae2c399af7363eaee25da333a6ac015d2aa3b33feee13dd77f1"
Jul 28 00:12:50.798414 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.797203    2570 scope.go:117] "RemoveContainer" containerID="8f1fdd2941e20f9d66652fcd1c5a3f5932235fb43074bcbda9fa634cbffa569e"
Jul 28 00:12:50.800862 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.800728    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"]
Jul 28 00:12:50.803592 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.803568    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-4a8122-6f766b5866-jm2k6"]
Jul 28 00:12:50.813471 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.813449    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"]
Jul 28 00:12:50.816971 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.816951    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-4a8122-5f986b9cd8-8nnzt"]
Jul 28 00:12:50.826758 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.826737    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"]
Jul 28 00:12:50.829746 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:50.829724    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4a8122-5b5b5ccd64-w274q"]
Jul 28 00:12:51.522164 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:51.522134    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b12240f-44aa-48a2-bad9-8f7fed96987b" path="/var/lib/kubelet/pods/4b12240f-44aa-48a2-bad9-8f7fed96987b/volumes"
Jul 28 00:12:51.522533 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:51.522520    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2c0064-2eff-4db3-944a-8f11f84ae59c" path="/var/lib/kubelet/pods/bf2c0064-2eff-4db3-944a-8f11f84ae59c/volumes"
Jul 28 00:12:51.522838 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:12:51.522827    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e523ad06-3470-421f-b533-0b023e108cb4" path="/var/lib/kubelet/pods/e523ad06-3470-421f-b533-0b023e108cb4/volumes"
Jul 28 00:13:06.642215 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:06.642125    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-559685848-vmr9g_68e30622-2f81-4f69-9c2e-4780d5b50551/manager/0.log"
Jul 28 00:13:07.111624 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:07.111530    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-ml926_41b05f88-317b-4659-bc01-44f6d991b868/manager/0.log"
Jul 28 00:13:07.117206 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:07.117189    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-ml926_41b05f88-317b-4659-bc01-44f6d991b868/copy-manifests/0.log"
Jul 28 00:13:07.572059 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:07.572028    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cbbdc4ff7-lzqqx_d94ec049-a86a-41b2-8fac-d5ecf4add937/maas-api/0.log"
Jul 28 00:13:07.784049 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:07.784023    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753265-hltt8_95646315-1948-47e6-933d-10572c1cfa25/cleanup/0.log"
Jul 28 00:13:10.418717 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:10.418678    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zzbzc_13739f21-ca0e-46c1-93b0-081b976f9ea5/discovery/0.log"
Jul 28 00:13:10.847779 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:10.847635    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2cf9t_76621e53-64f0-4420-96bd-178173721996/payload-pre-processing/0.log"
Jul 28 00:13:10.960840 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:10.960795    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mcpnn_b0d6a3ab-abed-40dd-a462-a521cbdef40e/payload-processing/0.log"
Jul 28 00:13:11.385552 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:11.385518    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-sghmx_e6f8763e-5292-41e7-b70e-c041e7a1abbe/main/0.log"
Jul 28 00:13:11.497175 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:11.497148    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kznlr_99b79ebe-4109-4a1c-8140-544be46f29c1/main/0.log"
Jul 28 00:13:19.690886 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:19.690852    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hvt44_8763dd4f-6c77-4916-88b7-01f454d22603/global-pull-secret-syncer/0.log"
Jul 28 00:13:19.815301 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:19.815268    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7ktzp_3debe752-2e41-4cd7-9265-80843637eb17/konnectivity-agent/0.log"
Jul 28 00:13:19.869810 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:19.869774    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-102.ec2.internal_76d1c0db1c0eb2e7cc88fd5e78b49f45/haproxy/0.log"
Jul 28 00:13:27.128910 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.128874    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/alertmanager/0.log"
Jul 28 00:13:27.152027 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.151994    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/config-reloader/0.log"
Jul 28 00:13:27.171636 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.171606    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/kube-rbac-proxy-web/0.log"
Jul 28 00:13:27.196627 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.196595    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/kube-rbac-proxy/0.log"
Jul 28 00:13:27.217934 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.217891    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/kube-rbac-proxy-metric/0.log"
Jul 28 00:13:27.238333 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.238301    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/prom-label-proxy/0.log"
Jul 28 00:13:27.258506 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.258477    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5b5d8d8-4640-49dd-a16b-489fd7fa0aa3/init-config-reloader/0.log"
Jul 28 00:13:27.333131 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.333101    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-954qf_d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd/kube-state-metrics/0.log"
Jul 28 00:13:27.347611 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.347577    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-954qf_d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd/kube-rbac-proxy-main/0.log"
Jul 28 00:13:27.367606 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.367578    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-954qf_d1a4ff3f-82f4-42f0-a54d-e447bcc8edbd/kube-rbac-proxy-self/0.log"
Jul 28 00:13:27.458605 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.458577    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sqsm_dc9dd18c-2d32-4756-b686-d3cf65fcce48/node-exporter/0.log"
Jul 28 00:13:27.478293 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.478265    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sqsm_dc9dd18c-2d32-4756-b686-d3cf65fcce48/kube-rbac-proxy/0.log"
Jul 28 00:13:27.497609 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.497580    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7sqsm_dc9dd18c-2d32-4756-b686-d3cf65fcce48/init-textfile/0.log"
Jul 28 00:13:27.974889 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.974858    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cb4fb8fd7-j4r7l_76418801-f504-450f-9bff-b67a129a62d9/telemeter-client/0.log"
Jul 28 00:13:27.993815 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:27.993784    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cb4fb8fd7-j4r7l_76418801-f504-450f-9bff-b67a129a62d9/reload/0.log"
Jul 28 00:13:28.012038 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:28.012018    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cb4fb8fd7-j4r7l_76418801-f504-450f-9bff-b67a129a62d9/kube-rbac-proxy/0.log"
Jul 28 00:13:29.861580 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:29.861536    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/1.log"
Jul 28 00:13:29.870184 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:29.870158    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tcw4n_6e7f7de0-e1cd-4854-90c2-b7c7c8ef28d7/console-operator/2.log"
Jul 28 00:13:30.047710 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.047673    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"]
Jul 28 00:13:30.048210 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048184    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e523ad06-3470-421f-b533-0b023e108cb4" containerName="payload-pre-processing"
Jul 28 00:13:30.048210 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048205    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e523ad06-3470-421f-b533-0b023e108cb4" containerName="payload-pre-processing"
Jul 28 00:13:30.048210 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048217    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048223    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e161adfa-1131-4054-884e-720b606cca79" containerName="istio-proxy"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048229    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de7a8e6-6b76-4c00-be63-be7411b6567f" containerName="revoke-keys"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048236    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de7a8e6-6b76-4c00-be63-be7411b6567f" containerName="revoke-keys"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048249    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6abc96b-6ad3-4d5b-9add-38349c43bd26" containerName="maas-api"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048254    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6abc96b-6ad3-4d5b-9add-38349c43bd26" containerName="maas-api"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048265    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e2931b-e84b-46bd-91f2-f52ff4697b31" containerName="payload-pre-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048271    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e2931b-e84b-46bd-91f2-f52ff4697b31" containerName="payload-pre-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048276    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c481186c-38f5-4aa7-a502-bc4f0a652044" containerName="payload-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048282    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c481186c-38f5-4aa7-a502-bc4f0a652044" containerName="payload-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048287    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048292    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048298    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b12240f-44aa-48a2-bad9-8f7fed96987b" containerName="payload-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048303    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b12240f-44aa-48a2-bad9-8f7fed96987b" containerName="payload-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048310    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2c0064-2eff-4db3-944a-8f11f84ae59c" containerName="maas-api"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048314    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2c0064-2eff-4db3-944a-8f11f84ae59c" containerName="maas-api"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048398    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c481186c-38f5-4aa7-a502-bc4f0a652044" containerName="payload-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048407    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2c0064-2eff-4db3-944a-8f11f84ae59c" containerName="maas-api"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048414    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de7a8e6-6b76-4c00-be63-be7411b6567f" containerName="revoke-keys"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048421    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ca41d0-f50e-4dce-aedd-cbc5c75b8c05" containerName="istio-proxy"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048426    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e2931b-e84b-46bd-91f2-f52ff4697b31" containerName="payload-pre-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048432    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b12240f-44aa-48a2-bad9-8f7fed96987b" containerName="payload-processing"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048437    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6abc96b-6ad3-4d5b-9add-38349c43bd26" containerName="maas-api"
Jul 28 00:13:30.048459 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.048442    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e523ad06-3470-421f-b533-0b023e108cb4" containerName="payload-pre-processing"
Jul 28 00:13:30.051484 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.051470    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.054449 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.054428    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lrsk\"/\"openshift-service-ca.crt\""
Jul 28 00:13:30.054577 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.054433    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9lrsk\"/\"default-dockercfg-d9m9z\""
Jul 28 00:13:30.054577 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.054482    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lrsk\"/\"kube-root-ca.crt\""
Jul 28 00:13:30.061975 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.061953    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"]
Jul 28 00:13:30.152299 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.152266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-sys\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.152528 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.152342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-podres\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.152528 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.152444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-lib-modules\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.152614 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.152538    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-proc\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.152614 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.152582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgddd\" (UniqueName: \"kubernetes.io/projected/e14f3e0b-60d9-4536-804c-9b73933dc9a3-kube-api-access-vgddd\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253565 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253522    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-proc\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253750 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgddd\" (UniqueName: \"kubernetes.io/projected/e14f3e0b-60d9-4536-804c-9b73933dc9a3-kube-api-access-vgddd\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253750 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253620    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-sys\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253750 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253648    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-proc\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253750 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253660    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-podres\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253750 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-lib-modules\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253750 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253736    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-sys\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253980 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253790    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-podres\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.253980 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.253803    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e14f3e0b-60d9-4536-804c-9b73933dc9a3-lib-modules\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.262096 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.262065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgddd\" (UniqueName: \"kubernetes.io/projected/e14f3e0b-60d9-4536-804c-9b73933dc9a3-kube-api-access-vgddd\") pod \"perf-node-gather-daemonset-ctwlt\" (UID: \"e14f3e0b-60d9-4536-804c-9b73933dc9a3\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.362491 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.362455    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:30.497435 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.497383    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"]
Jul 28 00:13:30.500406 ip-10-0-136-102 kubenswrapper[2570]: W0728 00:13:30.500384    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode14f3e0b_60d9_4536_804c_9b73933dc9a3.slice/crio-7fd58b863f22dd5411ff81df1446e80c50107834a53ccdb721620cbedfb45874 WatchSource:0}: Error finding container 7fd58b863f22dd5411ff81df1446e80c50107834a53ccdb721620cbedfb45874: Status 404 returned error can't find the container with id 7fd58b863f22dd5411ff81df1446e80c50107834a53ccdb721620cbedfb45874
Jul 28 00:13:30.986795 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.986755    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt" event={"ID":"e14f3e0b-60d9-4536-804c-9b73933dc9a3","Type":"ContainerStarted","Data":"db1b3e87d43143aa1672083cd2f4eb8a9943d838d4e20c30b8beccb9248e2a78"}
Jul 28 00:13:30.987224 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.986815    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt" event={"ID":"e14f3e0b-60d9-4536-804c-9b73933dc9a3","Type":"ContainerStarted","Data":"7fd58b863f22dd5411ff81df1446e80c50107834a53ccdb721620cbedfb45874"}
Jul 28 00:13:30.987224 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:30.986876    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:31.003317 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:31.003267    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt" podStartSLOduration=1.003252921 podStartE2EDuration="1.003252921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:13:31.001220024 +0000 UTC m=+3435.952563439" watchObservedRunningTime="2026-07-28 00:13:31.003252921 +0000 UTC m=+3435.954596308"
Jul 28 00:13:32.177906 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:32.177874    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zw7cg_fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c/dns/0.log"
Jul 28 00:13:32.196004 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:32.195972    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zw7cg_fc5f0f84-3c54-4f0b-a9e3-58318ffdb74c/kube-rbac-proxy/0.log"
Jul 28 00:13:32.239100 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:32.239062    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pc986_4a32c919-8faa-44fd-9e91-d696e209a766/dns-node-resolver/0.log"
Jul 28 00:13:32.799306 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:32.799272    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8799d4584-9dth5_b2d41c9a-ceac-4f55-85ce-45bf2b8427e9/registry/0.log"
Jul 28 00:13:32.843066 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:32.843028    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nkb9d_b48ab19b-f661-4ae8-84fe-96685d29fcfb/node-ca/0.log"
Jul 28 00:13:34.025802 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:34.025773    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zzbzc_13739f21-ca0e-46c1-93b0-081b976f9ea5/discovery/0.log"
Jul 28 00:13:34.161517 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:34.161485    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2cf9t_76621e53-64f0-4420-96bd-178173721996/payload-pre-processing/0.log"
Jul 28 00:13:34.196015 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:34.195984    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mcpnn_b0d6a3ab-abed-40dd-a462-a521cbdef40e/payload-processing/0.log"
Jul 28 00:13:34.993141 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:34.993097    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kbc6r_d5fb248c-397e-4373-9e50-8c22438a3fb1/serve-healthcheck-canary/0.log"
Jul 28 00:13:35.590325 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:35.590289    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-98hh2_9790621d-3b93-4f60-aba5-b4a771373ea9/kube-rbac-proxy/0.log"
Jul 28 00:13:35.608966 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:35.608939    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-98hh2_9790621d-3b93-4f60-aba5-b4a771373ea9/exporter/0.log"
Jul 28 00:13:35.628514 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:35.628452    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-98hh2_9790621d-3b93-4f60-aba5-b4a771373ea9/extractor/0.log"
Jul 28 00:13:37.001806 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:37.001776    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-ctwlt"
Jul 28 00:13:38.196676 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:38.196639    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-559685848-vmr9g_68e30622-2f81-4f69-9c2e-4780d5b50551/manager/0.log"
Jul 28 00:13:38.413163 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:38.413129    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-ml926_41b05f88-317b-4659-bc01-44f6d991b868/manager/0.log"
Jul 28 00:13:38.430820 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:38.430794    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-ml926_41b05f88-317b-4659-bc01-44f6d991b868/copy-manifests/0.log"
Jul 28 00:13:39.920675 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:39.920647    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6589476d7c-nnrpx_d4b2d777-5620-4cbf-b73e-67bc58956c4f/manager/0.log"
Jul 28 00:13:40.620819 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:40.620785    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zzbzc_13739f21-ca0e-46c1-93b0-081b976f9ea5/discovery/0.log"
Jul 28 00:13:40.722581 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:40.722542    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2cf9t_76621e53-64f0-4420-96bd-178173721996/payload-pre-processing/0.log"
Jul 28 00:13:40.750670 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:40.750621    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mcpnn_b0d6a3ab-abed-40dd-a462-a521cbdef40e/payload-processing/0.log"
Jul 28 00:13:42.497246 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:42.497189    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zzbzc_13739f21-ca0e-46c1-93b0-081b976f9ea5/discovery/0.log"
Jul 28 00:13:42.604741 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:42.604712    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-2cf9t_76621e53-64f0-4420-96bd-178173721996/payload-pre-processing/0.log"
Jul 28 00:13:42.633457 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:42.633422    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-mcpnn_b0d6a3ab-abed-40dd-a462-a521cbdef40e/payload-processing/0.log"
Jul 28 00:13:44.785285 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:44.785228    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-cjr89_53a03528-7f33-4d0e-9bad-2e2ffd268476/migrator/0.log"
Jul 28 00:13:44.803683 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:44.803651    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-cjr89_53a03528-7f33-4d0e-9bad-2e2ffd268476/graceful-termination/0.log"
Jul 28 00:13:45.133951 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:45.133851    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7_5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8/kube-storage-version-migrator-operator/1.log"
Jul 28 00:13:45.135460 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:45.135422    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-76tq7_5e8c2dee-8c8c-46e3-928d-fb8c42a78aa8/kube-storage-version-migrator-operator/0.log"
Jul 28 00:13:46.190974 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.190941    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-69bq9_d0d7cdaa-cf2d-4880-8a6d-24406bb2c7ea/kube-multus/0.log"
Jul 28 00:13:46.357658 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.357626    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/kube-multus-additional-cni-plugins/0.log"
Jul 28 00:13:46.376562 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.376536    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/egress-router-binary-copy/0.log"
Jul 28 00:13:46.395317 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.395289    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/cni-plugins/0.log"
Jul 28 00:13:46.413761 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.413738    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/bond-cni-plugin/0.log"
Jul 28 00:13:46.432282 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.432257    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/routeoverride-cni/0.log"
Jul 28 00:13:46.450817 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.450748    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/whereabouts-cni-bincopy/0.log"
Jul 28 00:13:46.469716 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.469692    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hdswt_72751d1b-30eb-40e1-b971-e83409fedf08/whereabouts-cni/0.log"
Jul 28 00:13:46.791129 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.791044    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vbgm4_b3acd34a-68bc-472f-a6e1-8169dbed08db/network-metrics-daemon/0.log"
Jul 28 00:13:46.804549 ip-10-0-136-102 kubenswrapper[2570]: I0728 00:13:46.804513    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vbgm4_b3acd34a-68bc-472f-a6e1-8169dbed08db/kube-rbac-proxy/0.log"