Sep 11 17:36:58.221268 ip-10-0-135-167 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 17:36:58.221280 ip-10-0-135-167 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 17:36:58.221287 ip-10-0-135-167 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 17:36:58.221608 ip-10-0-135-167 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 17:37:09.412894 ip-10-0-135-167 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 17:37:09.412909 ip-10-0-135-167 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a2dd864a7fc54c61962b2cc946262281 --
Sep 11 17:38:53.130801 ip-10-0-135-167 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:38:53.547302 ip-10-0-135-167 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:38:53.547974 ip-10-0-135-167 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:38:53.547974 ip-10-0-135-167 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:38:53.547974 ip-10-0-135-167 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:38:53.547974 ip-10-0-135-167 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:38:53.548716 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.548472    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:38:53.556721 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556705    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:38:53.556721 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556721    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556725    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556728    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556731    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556734    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556737    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556740    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556743    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556746    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556749    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556752    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556755    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556757    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556761    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556763    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556767    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556771    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556774    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:38:53.556794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556777    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556780    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556789    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556793    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556795    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556798    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556801    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556804    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556807    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556809    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556812    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556815    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556817    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556820    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556822    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556825    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556828    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556831    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556833    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556835    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:38:53.557222 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556838    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556841    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556843    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556846    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556848    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556850    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556853    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556855    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556857    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556860    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556863    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556867    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556869    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556872    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556875    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556878    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556880    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556883    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556886    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556888    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:38:53.557727 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556891    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556894    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556898    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556901    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556903    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556906    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556908    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556911    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556913    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556916    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556918    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556921    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556923    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556925    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556928    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556930    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556933    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556935    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556938    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:38:53.558248 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556941    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556946    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556950    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556953    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556955    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556957    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556960    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556964    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556966    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556969    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556972    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556974    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556977    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556979    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556982    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556984    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556987    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556989    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556992    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556995    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:38:53.558723 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.556997    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557701    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557710    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557713    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557716    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557720    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557722    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557725    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557728    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557731    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557734    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557737    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557739    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557742    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557746    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557750    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557754    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557756    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557759    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:38:53.559210 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557764    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557767    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557769    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557772    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557775    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557777    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557780    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557783    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557785    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557788    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557790    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557792    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557796    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557799    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557801    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557804    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557806    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557809    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557811    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557814    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:38:53.559695 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557816    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557819    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557821    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557824    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557826    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557829    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557831    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557834    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557837    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557840    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557842    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557845    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557849    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557852    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557855    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557858    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557862    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557865    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557868    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557871    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:38:53.560174 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557874    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557876    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557879    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557882    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557884    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557887    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557889    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557892    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557894    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557896    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557899    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557901    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557904    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557906    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557909    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557911    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557915    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557917    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557920    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557922    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:38:53.560672 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557925    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557929    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557931    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557934    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557938    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557941    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557944    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557946    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557948    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557951    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557953    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557956    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557958    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557961    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557963    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557965    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557968    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557970    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557973    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557975    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:38:53.561149 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.557978    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558349    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558354    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558357    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558361    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558365    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558368    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558371    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558374    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558377    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558379    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558382    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558385    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558388    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558391    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558393    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558397    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558400    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558403    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558405    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:38:53.561655 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558408    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558411    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558413    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558416    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558418    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558421    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558423    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558426    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558428    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558431    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558433    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558436    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558438    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558441    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558443    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558445    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558448    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558450    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558453    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558455    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:38:53.562141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558458    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558460    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558462    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558465    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558468    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558471    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558480    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558499    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558506    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558510    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558513    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558516    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558520    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558523    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558526    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558528    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558531    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558533    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558536    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:38:53.562653 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558538    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558541    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558544    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558546    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558549    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558551    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558554    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558556    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558559    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558561    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558564    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558566    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558569    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558571    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558574    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558576    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558579    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558582    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558584    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558587    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:38:53.563141 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558590    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558594    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558597    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558599    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558603    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558606    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558608    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558611    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558613    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558616    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558618    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558621    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558624    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558626    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558629    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558639    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558641    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558644    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558647    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558649    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:38:53.563646 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.558652    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558725    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558759    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558770    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558774    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558778    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558782    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558786    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558790    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558794    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558797    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558800    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558803    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558809    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558812    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558815    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558819    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558822    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558824    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558830    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558832    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558835    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558838    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558842    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:38:53.564130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558845    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558849    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558852    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558855    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558858    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558860    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558863    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558868    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558871    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558875    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558878    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558881    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558884    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558887    2563 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558889    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558894    2563 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558897    2563 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558899    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558902    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558905    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558909    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558913    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558916    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558919    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558922    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:38:53.564752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558925    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558928    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558931    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558934    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558937    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558940    2563 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558943    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558946    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558949    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558952    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558955    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558958    2563 flags.go:64] FLAG: --help="false"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558961    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558964    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558968    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558971    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558974    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558978    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558980    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558983    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558986    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558989    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558992    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558995    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:38:53.565370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.558998    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559000    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559003    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559007    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559010    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559013    2563 flags.go:64] FLAG: --lock-file=""
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559016    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559019    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559023    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559028    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559030    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559033    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559036    2563 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559039    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559042    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559045    2563 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559048    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559052    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559054    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559058    2563 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559061    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559064    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559070    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559072    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559075    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:38:53.566030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559078    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559081    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559088    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559091    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559094    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559097    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559100    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559106    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559109    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559112    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559115    2563 flags.go:64] FLAG: --port="10250"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559119    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559122    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03827a29e092c904c"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559125    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559128    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559131    2563 flags.go:64] FLAG: --register-node="true"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559134    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559137    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559140    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559143    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559146    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559149    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559153    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559155    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559158    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:38:53.566698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559161    2563 flags.go:64] FLAG: --runonce="false"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559164    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559167    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559170    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559172    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559176    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559179    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559182    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559185    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559188    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559191    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559194    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559197    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559200    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559203    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559205    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559211    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559214    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559218    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559222    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559225    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559228    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559239    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559243    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559246    2563 flags.go:64] FLAG: --v="2"
Sep 11 17:38:53.567407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559249    2563 flags.go:64] FLAG: --version="false"
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559253    2563 flags.go:64] FLAG: --vmodule=""
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559257    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.559260    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559385    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559389    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559392    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559395    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559399    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559401    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559404    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559407    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559411    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559413    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559416    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559418    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559421    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559424    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559427    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559431    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:38:53.568038 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559434    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559437    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559440    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559443    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559445    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559449    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559452    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559455    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559458    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559460    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559464    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559466    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559469    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559471    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559474    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559477    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559479    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559482    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559498    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559502    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:38:53.568539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559505    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559508    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559510    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559513    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559517    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559520    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559522    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559525    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559527    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559530    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559533    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559535    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559538    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559541    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559543    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559546    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559548    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559552    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559556    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:38:53.569024 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559559    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559561    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559564    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559567    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559575    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559577    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559580    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559582    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559585    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559587    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559590    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559592    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559595    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559597    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559600    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559602    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559605    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559610    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559612    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559615    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:38:53.569539 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559617    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559620    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559623    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559633    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559636    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559638    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559641    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559643    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559646    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559648    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559652    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559656    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559658    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559662    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559665    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559667    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559670    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559679    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559682    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559684    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:38:53.570072 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559687    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559690    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559693    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.559695    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.560441    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.568291    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.568307    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.569287    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.569298    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569358    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569365    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569368    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569372    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569375    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:38:53.570566 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569378    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569380    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569383    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569386    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569390    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569394    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569397    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569400    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569402    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569405    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569408    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569411    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569414    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569417    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569421    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569425    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569427    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569430    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569433    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:38:53.570922 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569436    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569439    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569441    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569444    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569446    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569449    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569452    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569455    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569457    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569460    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569463    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569465    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569468    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569470    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569473    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569475    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569478    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569481    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569496    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569499    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:38:53.571379 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569502    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569505    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569507    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569510    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569512    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569516    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569520    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569524    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569527    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569530    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569533    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569535    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569538    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569542    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569544    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569547    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569550    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569553    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569556    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:38:53.571884 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569558    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569561    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569564    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569566    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569569    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569572    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569574    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569577    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569580    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569583    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569586    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569589    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569591    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569594    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569597    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569599    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569602    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569604    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569607    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569609    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:38:53.572349 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569612    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569615    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569617    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569620    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569623    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569625    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569628    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569631    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569633    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569636    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569638    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569641    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569644    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569646    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569649    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:38:53.572846 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569651    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.569656    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569769    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569774    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569777    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569780    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569783    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569785    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569788    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569791    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569793    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569796    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569798    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569801    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569803    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:38:53.573205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569806    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569808    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569810    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569813    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569816    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569818    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569821    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569823    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569826    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569829    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569832    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569835    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569837    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569839    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569842    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569844    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569847    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569849    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569852    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:38:53.573598 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569854    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569857    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569859    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569861    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569864    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569866    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569869    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569872    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569874    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569876    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569879    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569881    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569884    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569886    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569889    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569891    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569894    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569896    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569899    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569901    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:38:53.574053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569904    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569907    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569910    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569914    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569917    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569920    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569924    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569926    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569929    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569932    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569935    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569937    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569940    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569942    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569945    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569947    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569950    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569952    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569955    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569958    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:38:53.574564 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569960    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569963    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569965    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569968    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569970    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569972    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569975    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569978    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569980    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569983    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569985    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569988    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569991    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569994    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.569998    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570002    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570005    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570007    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570010    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:38:53.575041 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570012    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570015    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570017    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570020    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570023    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570025    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570027    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:53.570030    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.570035    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.570132    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.572555    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.573742    2563 server.go:1019] "Starting client certificate rotation"
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.573843    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:38:53.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.574754    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:38:53.576584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.576570    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:38:53.578290 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.578272    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:38:53.588887 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.588866    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:38:53.592292 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.592276    2563 log.go:25] "Validated CRI v1 image API"
Sep 11 17:38:53.593140 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.593122    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:38:53.596462 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.596440    2563 fs.go:135] Filesystem UUIDs: map[1ae7a17b-1f55-4a05-9367-09885505f5c1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fce1d28d-705f-49bb-9997-a02375ee12eb:/dev/nvme0n1p4]
Sep 11 17:38:53.596576 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.596461    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:38:53.601837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.601716    2563 manager.go:216] Machine: {Timestamp:2026-09-11 17:38:53.600221421 +0000 UTC m=+0.363676142 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112286 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2660590b25adf5e1247d826f085df1 SystemUUID:ec266059-0b25-adf5-e124-7d826f085df1 BootID:a2dd864a-7fc5-4c61-962b-2cc946262281 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c5:7c:ff:89:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c5:7c:ff:89:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:84:aa:3e:f7:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:38:53.601837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.601828    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:38:53.602011 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.601940    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:38:53.605428 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.605404    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:38:53.605581 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.605430    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-167.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:38:53.606302 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.606291    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:38:53.606337 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.606307    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:38:53.606337 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.606314    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:38:53.606983 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.606972    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:38:53.608259 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.608249    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:38:53.613929 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.613917    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:38:53.614052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614042    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:38:53.614093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614084    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:38:53.614126 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614105    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:38:53.614126 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614120    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:38:53.614209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614129    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:38:53.614209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614138    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:38:53.614553 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614540    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:38:53.614593 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.614566    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:38:53.616994 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.616975    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:38:53.620467 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.620452    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:38:53.620552 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.620483    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:38:53.622389 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622376    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622393    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622399    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622405    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622414    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622419    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622432    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622437    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622444    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622450    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622459    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:38:53.622458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.622464    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:38:53.623442 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.623430    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:38:53.623442 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.623440    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:38:53.627095 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.627075    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:38:53.627237 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.627219    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:38:53.627306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.627289    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-167.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:38:53.627528 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.627515    2563 server.go:1268] "Started kubelet"
Sep 11 17:38:53.627599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.627576    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:38:53.627653 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.627604    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:38:53.627687 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.627681    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:38:53.628409 ip-10-0-135-167 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:38:53.629439 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.629243    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:38:53.629742 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.629722    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:38:53.629797 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.629746    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:38:53.633797 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.633764    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:38:53.636870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.636851    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:38:53.637630 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.637610    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:38:53.637729 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.637705    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:53.638404 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.638387    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:38:53.638524 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.638407    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:38:53.638524 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.638385    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:38:53.638624 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.638561    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:38:53.638624 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.638569    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:38:53.639182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639160    2563 factory.go:55] Registering systemd factory
Sep 11 17:38:53.639182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639181    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:38:53.639456 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639439    2563 factory.go:153] Registering CRI-O factory
Sep 11 17:38:53.639525 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639461    2563 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:38:53.639588 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639536    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:38:53.639588 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639557    2563 factory.go:103] Registering Raw factory
Sep 11 17:38:53.639588 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639571    2563 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:38:53.640007 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.639901    2563 manager.go:349] Starting recovery of all containers
Sep 11 17:38:53.649185 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.649150    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 17:38:53.650143 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.650118    2563 manager.go:354] Recovery completed
Sep 11 17:38:53.650472 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.649144    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1c58e618  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.627467288 +0000 UTC m=+0.390922006,LastTimestamp:2026-09-11 17:38:53.627467288 +0000 UTC m=+0.390922006,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.651832 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.651809    2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 11 17:38:53.654693 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.654679    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.656796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.656771    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.656862 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.656811    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.656862 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.656824    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.657528 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.657509    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.657608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.657535    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:38:53.657608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.657547    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:38:53.657608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.657564    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:38:53.658255 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.658241    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.658298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.658267    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.658298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.658277    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.659773 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.659761    2563 policy_none.go:49] "None policy: Start"
Sep 11 17:38:53.659829 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.659779    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:38:53.659829 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.659820    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:38:53.661794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.661780    2563 policy_none.go:47] "Start"
Sep 11 17:38:53.665228 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.665208    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-167.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 17:38:53.665320 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.665234    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1e186943  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-167.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.656795459 +0000 UTC m=+0.420250176,LastTimestamp:2026-09-11 17:38:53.656795459 +0000 UTC m=+0.420250176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.668118 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.668059    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1e18c1b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-167.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.656818097 +0000 UTC m=+0.420272815,LastTimestamp:2026-09-11 17:38:53.656818097 +0000 UTC m=+0.420272815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.678340 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.676817    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1e18e8c0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-167.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.656828096 +0000 UTC m=+0.420282814,LastTimestamp:2026-09-11 17:38:53.656828096 +0000 UTC m=+0.420282814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.689255 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.689190    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-167.ec2.internal.18d454ee1e186943\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1e186943  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-167.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.656795459 +0000 UTC m=+0.420250176,LastTimestamp:2026-09-11 17:38:53.65825488 +0000 UTC m=+0.421709598,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.700095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700072    2563 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.700119    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700132    2563 server.go:87] "Starting device plugin registration server"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700559    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700575    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700692    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700768    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.700776    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.701244    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.701279    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lk2tq"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.701297    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.703311    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-167.ec2.internal.18d454ee1e18c1b1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1e18c1b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-167.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.656818097 +0000 UTC m=+0.420272815,LastTimestamp:2026-09-11 17:38:53.65827155 +0000 UTC m=+0.421726269,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.717111 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.713083    2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-167.ec2.internal.18d454ee1e18e8c0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-167.ec2.internal.18d454ee1e18e8c0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-167.ec2.internal,UID:ip-10-0-135-167.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-167.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-167.ec2.internal,},FirstTimestamp:2026-09-11 17:38:53.656828096 +0000 UTC m=+0.420282814,LastTimestamp:2026-09-11 17:38:53.658281388 +0000 UTC m=+0.421736106,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-167.ec2.internal,}"
Sep 11 17:38:53.717776 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.717760    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lk2tq"
Sep 11 17:38:53.773379 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.773278    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:38:53.774616 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.774597    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:38:53.774681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.774675    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:38:53.774727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.774717    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:38:53.774782 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.774767    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:38:53.778680 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.778657    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:38:53.800906 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.800890    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.802535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.802520    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.802591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.802553    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.802591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.802566    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.802591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.802589    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.812976 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.812963    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.813034 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.812983    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-167.ec2.internal\": node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:53.838774 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.838749    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:53.875298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.875261    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"]
Sep 11 17:38:53.875421 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.875344    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.876789 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.876755    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.876789 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.876790    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.876912 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.876807    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.879254 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.879241    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.879377 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.879358    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.879410 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.879393    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.879957 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.879937    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.880050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.879971    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.880050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.879986    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.880050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.880013    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.880050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.880035    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.880050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.880049    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.882285 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.882271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.882365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.882293    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:38:53.882977 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.882960    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:38:53.883073 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.882991    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:38:53.883073 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.883007    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:38:53.897070 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.897051    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-167.ec2.internal\" not found" node="ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.900839 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.900822    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-167.ec2.internal\" not found" node="ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.939076 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:53.939044    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:53.940182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.940166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc5e2346b3de257dba80f041aa17c774-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"fc5e2346b3de257dba80f041aa17c774\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.940264 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.940195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc5e2346b3de257dba80f041aa17c774-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"fc5e2346b3de257dba80f041aa17c774\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:53.940264 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:53.940223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab0f818a53aaf7fb073af2872df2c0e1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-167.ec2.internal\" (UID: \"ab0f818a53aaf7fb073af2872df2c0e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.039406 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.039324    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.040466 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.040448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc5e2346b3de257dba80f041aa17c774-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"fc5e2346b3de257dba80f041aa17c774\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.040546 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.040478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc5e2346b3de257dba80f041aa17c774-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"fc5e2346b3de257dba80f041aa17c774\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.040546 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.040514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab0f818a53aaf7fb073af2872df2c0e1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-167.ec2.internal\" (UID: \"ab0f818a53aaf7fb073af2872df2c0e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.040644 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.040621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc5e2346b3de257dba80f041aa17c774-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"fc5e2346b3de257dba80f041aa17c774\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.040708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.040692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc5e2346b3de257dba80f041aa17c774-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal\" (UID: \"fc5e2346b3de257dba80f041aa17c774\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.040753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.040732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab0f818a53aaf7fb073af2872df2c0e1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-167.ec2.internal\" (UID: \"ab0f818a53aaf7fb073af2872df2c0e1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.139539 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.139478    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.199804 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.199769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.203273 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.203254    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.240034 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.240004    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.340620 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.340542    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.441044 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.441015    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.482934 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.482906    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:38:54.541453 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.541423    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.573724 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.573703    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:38:54.574138 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.573857    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:38:54.574138 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.573883    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:38:54.622345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.622274    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:38:54.642408 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:54.642389    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-167.ec2.internal\" not found"
Sep 11 17:38:54.718856 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.718828    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:38:54.723413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.723375    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:33:53 +0000 UTC" deadline="2028-06-13 19:21:46.274262068 +0000 UTC"
Sep 11 17:38:54.723413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.723412    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15385h42m51.550854223s"
Sep 11 17:38:54.738574 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.738547    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.750698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.750675    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:38:54.751436 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.751420    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal"
Sep 11 17:38:54.761168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.761144    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:38:54.820583 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:54.820541    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5e2346b3de257dba80f041aa17c774.slice/crio-cfbf5f2876af65ba4dc00f8583cf69e08ee04eadf4b5a087b35a16ffea35451c WatchSource:0}: Error finding container cfbf5f2876af65ba4dc00f8583cf69e08ee04eadf4b5a087b35a16ffea35451c: Status 404 returned error can't find the container with id cfbf5f2876af65ba4dc00f8583cf69e08ee04eadf4b5a087b35a16ffea35451c
Sep 11 17:38:54.820812 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:54.820793    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0f818a53aaf7fb073af2872df2c0e1.slice/crio-6c89eec37f8769f3669b3f2a24f5d4a26d9aa80b692bef38ef6cf7b83423e31c WatchSource:0}: Error finding container 6c89eec37f8769f3669b3f2a24f5d4a26d9aa80b692bef38ef6cf7b83423e31c: Status 404 returned error can't find the container with id 6c89eec37f8769f3669b3f2a24f5d4a26d9aa80b692bef38ef6cf7b83423e31c
Sep 11 17:38:54.824139 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:54.824124    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:38:55.615417 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.615385    2563 apiserver.go:52] "Watching apiserver"
Sep 11 17:38:55.623773 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.623743    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:38:55.625063 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.625028    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-57r2c","openshift-multus/multus-cg8dg","kube-system/global-pull-secret-syncer-94tdw","kube-system/konnectivity-agent-wpcq7","openshift-dns/node-resolver-kthkz","openshift-image-registry/node-ca-6mhjc","openshift-multus/network-metrics-daemon-fxqxq","openshift-network-diagnostics/network-check-target-9shmh","openshift-network-operator/iptables-alerter-x956r","openshift-ovn-kubernetes/ovnkube-node-9zj98","kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz","openshift-cluster-node-tuning-operator/tuned-f7qc2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal"]
Sep 11 17:38:55.627639 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.627619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:55.627740 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.627699    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:38:55.630017 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.629997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.632107 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.631990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.632107 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.632066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:38:55.634077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.634060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.634673 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.634656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tf97q\""
Sep 11 17:38:55.634855 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.634839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.634930 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.634874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.636762 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.636238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.637150 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.637131    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:38:55.638613 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.638593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:38:55.638706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.638694    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jk9gf\""
Sep 11 17:38:55.638767 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.638730    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.638820 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.638597    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:38:55.639045 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.639027    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.639112 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.639051    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vkpkh\""
Sep 11 17:38:55.639272 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.639255    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.641500 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.641466    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.642558 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.642539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.642773 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.642757    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lnz8n\""
Sep 11 17:38:55.642846 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.642773    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:38:55.642846 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.642823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.644696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.643862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.644696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.644058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8nnz8\""
Sep 11 17:38:55.644696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.644240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.644696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.644483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.644696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.644507    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:38:55.646024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.646006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.646560 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.646542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:38:55.647063 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647687    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647693    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysctl-d\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysctl-conf\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-run\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.647978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-tuned\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f2c6867-5f6f-4441-a673-1159fda5b848-agent-certs\") pod \"konnectivity-agent-wpcq7\" (UID: \"8f2c6867-5f6f-4441-a673-1159fda5b848\") " pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-kubernetes\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e38817a-672b-4e85-ab7c-405f8eb555c5-dbus\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648085    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glnbx\""
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f0836d0-c705-48f4-b420-cbada6b229a9-tmp-dir\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-tmp\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-modprobe-d\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-host\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4vh\" (UniqueName: \"kubernetes.io/projected/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-kube-api-access-kv4vh\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwcp\" (UniqueName: \"kubernetes.io/projected/7f0836d0-c705-48f4-b420-cbada6b229a9-kube-api-access-gxwcp\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c15f880-59bc-45ab-beff-5f7c3db77686-host\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-lib-modules\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctddc\" (UniqueName: \"kubernetes.io/projected/cf89e371-caa1-4d99-b78e-d1e2110f4266-kube-api-access-ctddc\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-sys\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwpl\" (UniqueName: \"kubernetes.io/projected/9c15f880-59bc-45ab-beff-5f7c3db77686-kube-api-access-xdwpl\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-var-lib-kubelet\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e38817a-672b-4e85-ab7c-405f8eb555c5-kubelet-config\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c6867-5f6f-4441-a673-1159fda5b848-konnectivity-ca\") pod \"konnectivity-agent-wpcq7\" (UID: \"8f2c6867-5f6f-4441-a673-1159fda5b848\") " pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c15f880-59bc-45ab-beff-5f7c3db77686-serviceca\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-systemd\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f0836d0-c705-48f4-b420-cbada6b229a9-hosts-file\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysconfig\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.648985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.648666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.650481 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.650460    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:38:55.650645 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.650620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tbbxd\""
Sep 11 17:38:55.650872 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.650853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:55.650941 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.650928    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:38:55.652031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652013    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:38:55.652122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:38:55.652269 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:38:55.652498 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652461    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.652587 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:38:55.652642 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652594    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.652804 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.652790    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xqr5d\""
Sep 11 17:38:55.653890 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.653871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.655349 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.655330    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:38:55.656742 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.656713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:38:55.657360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.657342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:38:55.657521 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.657505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:38:55.657679 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.657659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-48mhc\""
Sep 11 17:38:55.677200 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.677172    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xcjm"
Sep 11 17:38:55.688868 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.688843    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xcjm"
Sep 11 17:38:55.739637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.739601    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:38:55.748852 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.748820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4vh\" (UniqueName: \"kubernetes.io/projected/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-kube-api-access-kv4vh\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.748852 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.748863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-cni-netd\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.748889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqj7k\" (UniqueName: \"kubernetes.io/projected/359ad256-37e9-4293-b8a7-cb26ca499207-kube-api-access-nqj7k\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.749042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.748937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-cnibin\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.749042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.748975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-lib-modules\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-sys\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.749202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwpl\" (UniqueName: \"kubernetes.io/projected/9c15f880-59bc-45ab-beff-5f7c3db77686-kube-api-access-xdwpl\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.749202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-log-socket\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/359ad256-37e9-4293-b8a7-cb26ca499207-host-slash\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.749202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-lib-modules\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.749202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-registration-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-ovnkube-config\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-sys\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.749312    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-kubelet\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-systemd-units\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.749443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:56.249416999 +0000 UTC m=+3.012871725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:55.749516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-run-netns\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-etc-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-multus-certs\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-ovn\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-node-log\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-env-overrides\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysconfig\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-run\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f2c6867-5f6f-4441-a673-1159fda5b848-agent-certs\") pod \"konnectivity-agent-wpcq7\" (UID: \"8f2c6867-5f6f-4441-a673-1159fda5b848\") " pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-cni-bin\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-system-cni-dir\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-conf-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysconfig\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-run\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-tmp\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.749865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-slash\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1805cae-72e6-4e53-b586-3625d2c70393-ovn-node-metrics-cert\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-system-cni-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-modprobe-d\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-host\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.749990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwcp\" (UniqueName: \"kubernetes.io/projected/7f0836d0-c705-48f4-b420-cbada6b229a9-kube-api-access-gxwcp\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c15f880-59bc-45ab-beff-5f7c3db77686-host\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-systemd\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-var-lib-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-cnibin\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctddc\" (UniqueName: \"kubernetes.io/projected/cf89e371-caa1-4d99-b78e-d1e2110f4266-kube-api-access-ctddc\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-daemon-config\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-var-lib-kubelet\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.750706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e38817a-672b-4e85-ab7c-405f8eb555c5-kubelet-config\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c6867-5f6f-4441-a673-1159fda5b848-konnectivity-ca\") pod \"konnectivity-agent-wpcq7\" (UID: \"8f2c6867-5f6f-4441-a673-1159fda5b848\") " pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c15f880-59bc-45ab-beff-5f7c3db77686-serviceca\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750372    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e38817a-672b-4e85-ab7c-405f8eb555c5-kubelet-config\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-host\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c15f880-59bc-45ab-beff-5f7c3db77686-host\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-modprobe-d\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-var-lib-kubelet\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4rw\" (UniqueName: \"kubernetes.io/projected/c1805cae-72e6-4e53-b586-3625d2c70393-kube-api-access-pw4rw\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/58aed374-015c-4ef8-aae2-2409c4fb847c-kube-api-access-gzkhk\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-cni-multus\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c15f880-59bc-45ab-beff-5f7c3db77686-serviceca\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750902    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-socket-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-systemd\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.750975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-ovnkube-script-lib\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-netns\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f2c6867-5f6f-4441-a673-1159fda5b848-konnectivity-ca\") pod \"konnectivity-agent-wpcq7\" (UID: \"8f2c6867-5f6f-4441-a673-1159fda5b848\") " pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.751438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-systemd\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-cni-bin\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-etc-kubernetes\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qsd\" (UniqueName: \"kubernetes.io/projected/c5866eeb-3ea3-411b-802c-bf7623d682b9-kube-api-access-99qsd\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-hostroot\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f0836d0-c705-48f4-b420-cbada6b229a9-hosts-file\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysctl-d\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.751236    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysctl-conf\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.751287    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:56.251272183 +0000 UTC m=+3.014726889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-tuned\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-kubelet\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysctl-conf\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-socket-dir-parent\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-sys-fs\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.752243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-kubernetes\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f0836d0-c705-48f4-b420-cbada6b229a9-hosts-file\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e38817a-672b-4e85-ab7c-405f8eb555c5-dbus\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-kubernetes\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f0836d0-c705-48f4-b420-cbada6b229a9-tmp-dir\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-sysctl-d\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-cni-binary-copy\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751647    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e38817a-672b-4e85-ab7c-405f8eb555c5-dbus\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-cni-binary-copy\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/359ad256-37e9-4293-b8a7-cb26ca499207-iptables-alerter-script\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-cni-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-k8s-cni-cncf-io\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mmg\" (UniqueName: \"kubernetes.io/projected/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-kube-api-access-m7mmg\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-device-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.753024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.751837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f0836d0-c705-48f4-b420-cbada6b229a9-tmp-dir\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.755508 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.754795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-etc-tuned\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.755508 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.754853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-tmp\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.755508 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.755111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f2c6867-5f6f-4441-a673-1159fda5b848-agent-certs\") pod \"konnectivity-agent-wpcq7\" (UID: \"8f2c6867-5f6f-4441-a673-1159fda5b848\") " pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.761054 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.761029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4vh\" (UniqueName: \"kubernetes.io/projected/8a5df42c-1a4d-4543-a6b5-d303d775e5ff-kube-api-access-kv4vh\") pod \"tuned-f7qc2\" (UID: \"8a5df42c-1a4d-4543-a6b5-d303d775e5ff\") " pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.761929 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.761911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwcp\" (UniqueName: \"kubernetes.io/projected/7f0836d0-c705-48f4-b420-cbada6b229a9-kube-api-access-gxwcp\") pod \"node-resolver-kthkz\" (UID: \"7f0836d0-c705-48f4-b420-cbada6b229a9\") " pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.762421 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.762394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctddc\" (UniqueName: \"kubernetes.io/projected/cf89e371-caa1-4d99-b78e-d1e2110f4266-kube-api-access-ctddc\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:55.762762 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.762748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwpl\" (UniqueName: \"kubernetes.io/projected/9c15f880-59bc-45ab-beff-5f7c3db77686-kube-api-access-xdwpl\") pod \"node-ca-6mhjc\" (UID: \"9c15f880-59bc-45ab-beff-5f7c3db77686\") " pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.780565 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.780482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" event={"ID":"fc5e2346b3de257dba80f041aa17c774","Type":"ContainerStarted","Data":"cfbf5f2876af65ba4dc00f8583cf69e08ee04eadf4b5a087b35a16ffea35451c"}
Sep 11 17:38:55.784104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.784069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal" event={"ID":"ab0f818a53aaf7fb073af2872df2c0e1","Type":"ContainerStarted","Data":"6c89eec37f8769f3669b3f2a24f5d4a26d9aa80b692bef38ef6cf7b83423e31c"}
Sep 11 17:38:55.852923 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.852888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/359ad256-37e9-4293-b8a7-cb26ca499207-iptables-alerter-script\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.852923 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.852925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-cni-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.852947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-k8s-cni-cncf-io\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.852966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mmg\" (UniqueName: \"kubernetes.io/projected/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-kube-api-access-m7mmg\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.852986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-device-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-cni-netd\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqj7k\" (UniqueName: \"kubernetes.io/projected/359ad256-37e9-4293-b8a7-cb26ca499207-kube-api-access-nqj7k\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-cnibin\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-log-socket\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-cni-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.853147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/359ad256-37e9-4293-b8a7-cb26ca499207-host-slash\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/359ad256-37e9-4293-b8a7-cb26ca499207-host-slash\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-registration-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-ovnkube-config\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-kubelet\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-systemd-units\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-run-netns\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-etc-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-multus-certs\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-ovn\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-node-log\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-env-overrides\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/359ad256-37e9-4293-b8a7-cb26ca499207-iptables-alerter-script\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-cni-bin\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-system-cni-dir\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.853595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-conf-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-slash\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1805cae-72e6-4e53-b586-3625d2c70393-ovn-node-metrics-cert\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-system-cni-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-systemd\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-var-lib-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-cnibin\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-cni-netd\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-k8s-cni-cncf-io\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-daemon-config\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4rw\" (UniqueName: \"kubernetes.io/projected/c1805cae-72e6-4e53-b586-3625d2c70393-kube-api-access-pw4rw\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.854366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/58aed374-015c-4ef8-aae2-2409c4fb847c-kube-api-access-gzkhk\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-cni-multus\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-ovnkube-config\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.853983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-socket-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-ovnkube-script-lib\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-cnibin\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-netns\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-log-socket\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-device-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-cni-bin\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-registration-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-kubelet\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-cni-bin\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-etc-kubernetes\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qsd\" (UniqueName: \"kubernetes.io/projected/c5866eeb-3ea3-411b-802c-bf7623d682b9-kube-api-access-99qsd\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-systemd\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-system-cni-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.855202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-hostroot\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-cnibin\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-kubelet\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-socket-dir-parent\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-run-ovn-kubernetes\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-sys-fs\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-cni-binary-copy\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-cni-binary-copy\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-env-overrides\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-var-lib-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-systemd-units\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-run-netns\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-etc-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-multus-certs\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-ovn\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-node-log\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-var-lib-cni-multus\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.854958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-cni-binary-copy\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-cni-bin\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-system-cni-dir\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-conf-dir\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-socket-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-slash\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-daemon-config\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-etc-kubernetes\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1805cae-72e6-4e53-b586-3625d2c70393-ovnkube-script-lib\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-host-kubelet\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-host-run-netns\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-hostroot\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-multus-socket-dir-parent\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.856617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1805cae-72e6-4e53-b586-3625d2c70393-run-openvswitch\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.857169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c5866eeb-3ea3-411b-802c-bf7623d682b9-sys-fs\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.857169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.855995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58aed374-015c-4ef8-aae2-2409c4fb847c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.857169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.856296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58aed374-015c-4ef8-aae2-2409c4fb847c-cni-binary-copy\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.857964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.857947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1805cae-72e6-4e53-b586-3625d2c70393-ovn-node-metrics-cert\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.876938 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.876381    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:38:55.876938 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.876427    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:38:55.876938 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.876541    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:55.876938 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:55.876681    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:56.376663311 +0000 UTC m=+3.140118035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:55.878978 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.878581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkhk\" (UniqueName: \"kubernetes.io/projected/58aed374-015c-4ef8-aae2-2409c4fb847c-kube-api-access-gzkhk\") pod \"multus-additional-cni-plugins-57r2c\" (UID: \"58aed374-015c-4ef8-aae2-2409c4fb847c\") " pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.878978 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.878853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqj7k\" (UniqueName: \"kubernetes.io/projected/359ad256-37e9-4293-b8a7-cb26ca499207-kube-api-access-nqj7k\") pod \"iptables-alerter-x956r\" (UID: \"359ad256-37e9-4293-b8a7-cb26ca499207\") " pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.878978 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.878944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qsd\" (UniqueName: \"kubernetes.io/projected/c5866eeb-3ea3-411b-802c-bf7623d682b9-kube-api-access-99qsd\") pod \"aws-ebs-csi-driver-node-dv4mz\" (UID: \"c5866eeb-3ea3-411b-802c-bf7623d682b9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:55.879197 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.879174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mmg\" (UniqueName: \"kubernetes.io/projected/7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2-kube-api-access-m7mmg\") pod \"multus-cg8dg\" (UID: \"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2\") " pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:55.879326 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.879311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4rw\" (UniqueName: \"kubernetes.io/projected/c1805cae-72e6-4e53-b586-3625d2c70393-kube-api-access-pw4rw\") pod \"ovnkube-node-9zj98\" (UID: \"c1805cae-72e6-4e53-b586-3625d2c70393\") " pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:55.941176 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.941141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7qc2"
Sep 11 17:38:55.953377 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.953356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:38:55.962080 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.962058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kthkz"
Sep 11 17:38:55.975662 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.975629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6mhjc"
Sep 11 17:38:55.982411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.982380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x956r"
Sep 11 17:38:55.990051 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.990031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57r2c"
Sep 11 17:38:55.997727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:55.997705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cg8dg"
Sep 11 17:38:56.005452 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.005433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:38:56.011094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.011077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz"
Sep 11 17:38:56.053967 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.053929    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:38:56.258843 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.258748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:56.258843 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.258809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:56.259042 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.258903    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:56.259042 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.258971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:57.258951284 +0000 UTC m=+4.022405989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:56.259042 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.258903    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:56.259201 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.259072    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:57.259046476 +0000 UTC m=+4.022501204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:56.459920 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.459848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:56.460090 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.460053    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:38:56.460090 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.460083    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:38:56.460173 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.460098    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:56.460173 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:56.460166    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:57.460146782 +0000 UTC m=+4.223601505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:56.692197 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.692162    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:33:55 +0000 UTC" deadline="2028-02-25 03:25:15.890787741 +0000 UTC"
Sep 11 17:38:56.692470 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.692197    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12753h46m19.198595111s"
Sep 11 17:38:56.700026 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.699986    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359ad256_37e9_4293_b8a7_cb26ca499207.slice/crio-b83c9fac77e57c5de11a8bd28f002e9c5d18713a0cdce689edabfec93e13ea42 WatchSource:0}: Error finding container b83c9fac77e57c5de11a8bd28f002e9c5d18713a0cdce689edabfec93e13ea42: Status 404 returned error can't find the container with id b83c9fac77e57c5de11a8bd28f002e9c5d18713a0cdce689edabfec93e13ea42
Sep 11 17:38:56.702869 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.702843    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a181b1a_0e9b_4e60_a88a_f6eb071ba6f2.slice/crio-1c253c28d0c023d3dcac5844f3f38ef5dadb13f76ceeabaf71abe1f20d79fb64 WatchSource:0}: Error finding container 1c253c28d0c023d3dcac5844f3f38ef5dadb13f76ceeabaf71abe1f20d79fb64: Status 404 returned error can't find the container with id 1c253c28d0c023d3dcac5844f3f38ef5dadb13f76ceeabaf71abe1f20d79fb64
Sep 11 17:38:56.707277 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.707252    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2c6867_5f6f_4441_a673_1159fda5b848.slice/crio-4bf97a5d1f1a188d42d558fb5cb6e4cda7c5f8839c8128602c91b50640fb17cf WatchSource:0}: Error finding container 4bf97a5d1f1a188d42d558fb5cb6e4cda7c5f8839c8128602c91b50640fb17cf: Status 404 returned error can't find the container with id 4bf97a5d1f1a188d42d558fb5cb6e4cda7c5f8839c8128602c91b50640fb17cf
Sep 11 17:38:56.709053 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.709028    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5866eeb_3ea3_411b_802c_bf7623d682b9.slice/crio-10b7e4aca9ee769113c3a96fc411a901ac894c4e63af547ea9efe4d993b8a116 WatchSource:0}: Error finding container 10b7e4aca9ee769113c3a96fc411a901ac894c4e63af547ea9efe4d993b8a116: Status 404 returned error can't find the container with id 10b7e4aca9ee769113c3a96fc411a901ac894c4e63af547ea9efe4d993b8a116
Sep 11 17:38:56.710011 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.709976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aed374_015c_4ef8_aae2_2409c4fb847c.slice/crio-df148b7491c6667e04298cc033d6971d47e1fd5904749ff70fbd924f670bdd11 WatchSource:0}: Error finding container df148b7491c6667e04298cc033d6971d47e1fd5904749ff70fbd924f670bdd11: Status 404 returned error can't find the container with id df148b7491c6667e04298cc033d6971d47e1fd5904749ff70fbd924f670bdd11
Sep 11 17:38:56.710415 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.710397    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0836d0_c705_48f4_b420_cbada6b229a9.slice/crio-aa8b6ee1f540593a59b94bd8c686bbe43d547b1a53f9997138ddb0c9c947d464 WatchSource:0}: Error finding container aa8b6ee1f540593a59b94bd8c686bbe43d547b1a53f9997138ddb0c9c947d464: Status 404 returned error can't find the container with id aa8b6ee1f540593a59b94bd8c686bbe43d547b1a53f9997138ddb0c9c947d464
Sep 11 17:38:56.712548 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:38:56.712344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1805cae_72e6_4e53_b586_3625d2c70393.slice/crio-359bfa79bccb623ed787b129c233df71c1b56ab1ae9b82b592766dbb3decbbf7 WatchSource:0}: Error finding container 359bfa79bccb623ed787b129c233df71c1b56ab1ae9b82b592766dbb3decbbf7: Status 404 returned error can't find the container with id 359bfa79bccb623ed787b129c233df71c1b56ab1ae9b82b592766dbb3decbbf7
Sep 11 17:38:56.787582 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.787556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7qc2" event={"ID":"8a5df42c-1a4d-4543-a6b5-d303d775e5ff","Type":"ContainerStarted","Data":"c84cdf88d2769c56b3ebde80d80e2d717315cc83482afb23a9c0cf83d0f66aa1"}
Sep 11 17:38:56.788689 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.788669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"359bfa79bccb623ed787b129c233df71c1b56ab1ae9b82b592766dbb3decbbf7"}
Sep 11 17:38:56.789928 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.789904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kthkz" event={"ID":"7f0836d0-c705-48f4-b420-cbada6b229a9","Type":"ContainerStarted","Data":"aa8b6ee1f540593a59b94bd8c686bbe43d547b1a53f9997138ddb0c9c947d464"}
Sep 11 17:38:56.791104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.791083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerStarted","Data":"df148b7491c6667e04298cc033d6971d47e1fd5904749ff70fbd924f670bdd11"}
Sep 11 17:38:56.792232 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.792208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x956r" event={"ID":"359ad256-37e9-4293-b8a7-cb26ca499207","Type":"ContainerStarted","Data":"b83c9fac77e57c5de11a8bd28f002e9c5d18713a0cdce689edabfec93e13ea42"}
Sep 11 17:38:56.793272 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.793249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz" event={"ID":"c5866eeb-3ea3-411b-802c-bf7623d682b9","Type":"ContainerStarted","Data":"10b7e4aca9ee769113c3a96fc411a901ac894c4e63af547ea9efe4d993b8a116"}
Sep 11 17:38:56.794888 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.794866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6mhjc" event={"ID":"9c15f880-59bc-45ab-beff-5f7c3db77686","Type":"ContainerStarted","Data":"be946b36c4998081c124765cee77c0991fea4339522debbf3d13588ed6b9371f"}
Sep 11 17:38:56.796074 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.796052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wpcq7" event={"ID":"8f2c6867-5f6f-4441-a673-1159fda5b848","Type":"ContainerStarted","Data":"4bf97a5d1f1a188d42d558fb5cb6e4cda7c5f8839c8128602c91b50640fb17cf"}
Sep 11 17:38:56.797412 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.797389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cg8dg" event={"ID":"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2","Type":"ContainerStarted","Data":"1c253c28d0c023d3dcac5844f3f38ef5dadb13f76ceeabaf71abe1f20d79fb64"}
Sep 11 17:38:56.864220 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:56.864169    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:38:57.266194 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.266158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:57.266388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.266223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:57.266388 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.266324    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:57.266388 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.266338    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:57.266388 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.266383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:59.266368983 +0000 UTC m=+6.029823702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:57.266639 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.266426    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:59.266408124 +0000 UTC m=+6.029862844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:57.468023 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.467984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:57.468193 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.468147    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:38:57.468193 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.468168    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:38:57.468193 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.468181    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:57.468356 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.468264    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:38:59.468219951 +0000 UTC m=+6.231674661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:57.693259 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.693143    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:33:55 +0000 UTC" deadline="2028-06-17 21:47:56.859308112 +0000 UTC"
Sep 11 17:38:57.693259 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.693208    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15484h8m59.166104081s"
Sep 11 17:38:57.776809 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.776775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:57.776987 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.776914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:38:57.777047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.777004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:57.777144 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.777119    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:38:57.777298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.777259    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:57.777406 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:57.777366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:38:57.817649 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.817611    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc5e2346b3de257dba80f041aa17c774" containerID="158fa7f5ea8daa3ab5117ab5ecb7df8823931369b4f21ac330a1cc21a54987e6" exitCode=0
Sep 11 17:38:57.817883 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.817707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" event={"ID":"fc5e2346b3de257dba80f041aa17c774","Type":"ContainerDied","Data":"158fa7f5ea8daa3ab5117ab5ecb7df8823931369b4f21ac330a1cc21a54987e6"}
Sep 11 17:38:57.836409 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.836328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal" event={"ID":"ab0f818a53aaf7fb073af2872df2c0e1","Type":"ContainerStarted","Data":"7e954fc2b8192a77fad8a2638ef70e531c60c783f33ae9904d33824ae573af63"}
Sep 11 17:38:57.852682 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:57.852227    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-167.ec2.internal" podStartSLOduration=3.852205567 podStartE2EDuration="3.852205567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:57.8515667 +0000 UTC m=+4.615021427" watchObservedRunningTime="2026-09-11 17:38:57.852205567 +0000 UTC m=+4.615660297"
Sep 11 17:38:58.856413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:58.856131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" event={"ID":"fc5e2346b3de257dba80f041aa17c774","Type":"ContainerStarted","Data":"24d89de63cec2a469d8ca73c8c2ddea8cd05b9422feaf84e1fb1e4aa2386ad31"}
Sep 11 17:38:58.873113 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:58.873055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-167.ec2.internal" podStartSLOduration=4.873033154 podStartE2EDuration="4.873033154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:58.872427364 +0000 UTC m=+5.635882094" watchObservedRunningTime="2026-09-11 17:38:58.873033154 +0000 UTC m=+5.636487882"
Sep 11 17:38:59.293007 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:59.292166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:59.293007 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:59.292235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:59.293007 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.292383    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:59.293007 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.292447    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:03.292428439 +0000 UTC m=+10.055883152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:38:59.293007 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.292872    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:59.293007 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.292923    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:03.29290732 +0000 UTC m=+10.056362027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:38:59.493650 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:59.493608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:59.493817 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.493775    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:38:59.493817 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.493794    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:38:59.493817 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.493808    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:59.494003 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.493867    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:03.493848668 +0000 UTC m=+10.257303395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:38:59.775699 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:59.775602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:38:59.775863 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.775734    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:38:59.776164 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:59.776146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:38:59.776292 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.776243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:38:59.778529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:38:59.778505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:38:59.778641 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:38:59.778618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:01.777744 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:01.776942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:01.777744 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:01.777087    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:01.777744 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:01.777154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:01.777744 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:01.777221    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:01.777744 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:01.777261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:01.777744 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:01.777323    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:03.329217 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:03.328914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:03.329217 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:03.329071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:03.329217 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.329087    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:39:03.329217 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.329154    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:11.329134303 +0000 UTC m=+18.092589052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:39:03.329217 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.329159    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:39:03.329217 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.329192    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:11.329181833 +0000 UTC m=+18.092636552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:39:03.530724 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:03.530686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:03.531049 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.531012    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:39:03.531174 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.531058    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:39:03.531174 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.531073    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:39:03.531174 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.531145    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:11.531128605 +0000 UTC m=+18.294583330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:39:03.777501 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:03.777412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:03.777660 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.777560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:03.777660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:03.777595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:03.777765 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.777666    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:03.777765 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:03.777667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:03.778779 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:03.778743    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:05.775602 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:05.774935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:05.775602 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:05.775064    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:05.775602 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:05.774935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:05.775602 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:05.775169    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:05.775602 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:05.774935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:05.775602 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:05.775243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:07.775916 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:07.775872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:07.776360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:07.775873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:07.776360 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:07.776016    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:07.776360 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:07.776107    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:07.776360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:07.775873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:07.776360 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:07.776207    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:09.775328 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:09.775287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:09.775328 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:09.775324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:09.775884 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:09.775287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:09.775884 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:09.775431    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:09.775884 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:09.775518    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:09.775884 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:09.775598    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:11.388639 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:11.388592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:11.389084 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:11.388663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:11.389084 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.388786    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:39:11.389084 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.388870    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:27.388848469 +0000 UTC m=+34.152303176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:39:11.389084 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.388791    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:39:11.389084 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.388943    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:27.388929402 +0000 UTC m=+34.152384123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:39:11.590176 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:11.590136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:11.590361 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.590280    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:39:11.590361 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.590308    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:39:11.590361 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.590321    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:39:11.590536 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.590389    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:27.590371324 +0000 UTC m=+34.353826046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:39:11.775652 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:11.775578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:11.775652 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:11.775598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:11.775862 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.775702    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:11.775862 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:11.775756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:11.775862 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.775851    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:11.776020 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:11.775950    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:13.775677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:13.775646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:13.776161 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:13.775765    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:13.776161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:13.775820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:13.776161 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:13.775941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:13.776161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:13.775994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:13.776161 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:13.776115    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:14.887028 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.886798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wpcq7" event={"ID":"8f2c6867-5f6f-4441-a673-1159fda5b848","Type":"ContainerStarted","Data":"67b309baf5498c0fda6b451ba506809083b1fdb3dd3c46479612216d9d7f925d"}
Sep 11 17:39:14.894098 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.894023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cg8dg" event={"ID":"7a181b1a-0e9b-4e60-a88a-f6eb071ba6f2","Type":"ContainerStarted","Data":"be1de9110046ed09bece92865867a91df5812e08433eae436306ac60bad5a7da"}
Sep 11 17:39:14.895852 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.895823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7qc2" event={"ID":"8a5df42c-1a4d-4543-a6b5-d303d775e5ff","Type":"ContainerStarted","Data":"ddc34957f4b338283b99faf51db4387769e147189c00e7a6108e174e1620befd"}
Sep 11 17:39:14.897836 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.897812    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:39:14.898275 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.898247    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1805cae-72e6-4e53-b586-3625d2c70393" containerID="d3ea30ab1a5ae3dfd56767da4cfab039d823de4e8ca0cd81333a69314b02f812" exitCode=1
Sep 11 17:39:14.898379 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.898287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerDied","Data":"d3ea30ab1a5ae3dfd56767da4cfab039d823de4e8ca0cd81333a69314b02f812"}
Sep 11 17:39:14.898379 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.898372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"cdb5a281a20fcb835a5bd79d322e71ecaaec7458708f588efae004f629b757cb"}
Sep 11 17:39:14.900230 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.900206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kthkz" event={"ID":"7f0836d0-c705-48f4-b420-cbada6b229a9","Type":"ContainerStarted","Data":"d2f62016af371238fac66625dff29a4088478c972998d0d0fd324a71e1dd9ac4"}
Sep 11 17:39:14.901978 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.901947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerStarted","Data":"266befdb5a333bdbb45c83848db2a40da290329289c6273cb9a6124af4aba647"}
Sep 11 17:39:14.903721 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.903696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz" event={"ID":"c5866eeb-3ea3-411b-802c-bf7623d682b9","Type":"ContainerStarted","Data":"5593a7192d8b8a7e2b8ccc44fdbe36ba54f926d8b1d211e73ac0dc200f6a1fc6"}
Sep 11 17:39:14.905322 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.905299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6mhjc" event={"ID":"9c15f880-59bc-45ab-beff-5f7c3db77686","Type":"ContainerStarted","Data":"16bd07b905527d6aa46c48b22a908211bbb53e4b7e15a5d490584810c4a334e9"}
Sep 11 17:39:14.905727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.905677    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wpcq7" podStartSLOduration=8.399675485 podStartE2EDuration="21.905658306s" totalImagesPullingTime="13.505982821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:14.904856093 +0000 UTC m=+21.668310822" watchObservedRunningTime="2026-09-11 17:39:14.905658306 +0000 UTC m=+21.669113128"
Sep 11 17:39:14.922839 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.922775    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f7qc2" podStartSLOduration=4.315263767 podStartE2EDuration="21.922750941s" totalImagesPullingTime="17.607487174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:14.922388059 +0000 UTC m=+21.685842786" watchObservedRunningTime="2026-09-11 17:39:14.922750941 +0000 UTC m=+21.686205685"
Sep 11 17:39:14.967155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:14.967101    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kthkz" podStartSLOduration=4.372439699 podStartE2EDuration="21.967077317s" totalImagesPullingTime="17.594637618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:14.937082652 +0000 UTC m=+21.700537391" watchObservedRunningTime="2026-09-11 17:39:14.967077317 +0000 UTC m=+21.730532045"
Sep 11 17:39:15.005887 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.005835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6mhjc" podStartSLOduration=4.419124157 podStartE2EDuration="22.005818239s" totalImagesPullingTime="17.586694082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:15.00572906 +0000 UTC m=+21.769183783" watchObservedRunningTime="2026-09-11 17:39:15.005818239 +0000 UTC m=+21.769272966"
Sep 11 17:39:15.006314 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.006293    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cg8dg" podStartSLOduration=4.337611249 podStartE2EDuration="22.006286382s" totalImagesPullingTime="17.668675133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:14.990406688 +0000 UTC m=+21.753861415" watchObservedRunningTime="2026-09-11 17:39:15.006286382 +0000 UTC m=+21.769741111"
Sep 11 17:39:15.775257 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.775213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:15.775257 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.775260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:15.775582 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.775232    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:15.775582 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:15.775329    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:15.775582 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:15.775389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:15.775582 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:15.775449    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:15.910160 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.910131    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:39:15.910881 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.910483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"4b38139887eda85685890b9a4aacc619cb377dae038602cb71f52fd44d4e282e"}
Sep 11 17:39:15.910881 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.910577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"3147df33ab316ce8e7f0417a83b66b4683d3f7ff003c456111d1462b122ac4c4"}
Sep 11 17:39:15.910881 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.910629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"19dd23fa01def212dec0c12c79c5494bb7c4dffbf0b608d88361472bc84a2634"}
Sep 11 17:39:15.910881 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.910679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"68e06ece497666e83dcc4a419e180551f63a7310359a73fd4362f86751b9a2a1"}
Sep 11 17:39:15.913726 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.913702    2563 generic.go:356] "Generic (PLEG): container finished" podID="58aed374-015c-4ef8-aae2-2409c4fb847c" containerID="266befdb5a333bdbb45c83848db2a40da290329289c6273cb9a6124af4aba647" exitCode=0
Sep 11 17:39:15.913857 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.913763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerDied","Data":"266befdb5a333bdbb45c83848db2a40da290329289c6273cb9a6124af4aba647"}
Sep 11 17:39:15.915144 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.915113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x956r" event={"ID":"359ad256-37e9-4293-b8a7-cb26ca499207","Type":"ContainerStarted","Data":"706964fca86a642b455b28efea8ca7c58191d6c915480f6c9113584a6720e66d"}
Sep 11 17:39:15.949244 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:15.949190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x956r" podStartSLOduration=5.345619997 podStartE2EDuration="22.949174554s" totalImagesPullingTime="17.603554557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:15.94899871 +0000 UTC m=+22.712453438" watchObservedRunningTime="2026-09-11 17:39:15.949174554 +0000 UTC m=+22.712629294"
Sep 11 17:39:16.146230 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:16.146067    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:39:16.713303 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:16.713198    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:39:16.146225466Z","UUID":"39bbac07-d103-43a4-beae-f20123b1ae9c","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:39:16.714958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:16.714930    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:39:16.714958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:16.714966    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:39:16.921016 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:16.920983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz" event={"ID":"c5866eeb-3ea3-411b-802c-bf7623d682b9","Type":"ContainerStarted","Data":"861a752bdc503ce34cf3af2c7a7eedabd943b34596b8410947e591ef2c2f415e"}
Sep 11 17:39:17.775093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.775007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:17.775093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.775033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:17.775287 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.775007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:17.775287 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:17.775128    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:17.775287 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:17.775221    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:17.775403 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:17.775302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:17.925752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.925713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz" event={"ID":"c5866eeb-3ea3-411b-802c-bf7623d682b9","Type":"ContainerStarted","Data":"acf3ce3b5cf5b93e9c38dbb964c411e0de76fda7b8afd1822818057670f8e8db"}
Sep 11 17:39:17.929314 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.929288    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:39:17.929860 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.929835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"3433e88e437ff1155caed7ddd961543e152f6ad975afccd6581cf3ae5e73ca7e"}
Sep 11 17:39:17.945447 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:17.945388    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dv4mz" podStartSLOduration=4.7267399690000005 podStartE2EDuration="24.945368658s" totalImagesPullingTime="20.218628689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:17.944966874 +0000 UTC m=+24.708421624" watchObservedRunningTime="2026-09-11 17:39:17.945368658 +0000 UTC m=+24.708823387"
Sep 11 17:39:18.406700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:18.406663    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:39:18.407339 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:18.407319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:39:19.775182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:19.775149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:19.775182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:19.775173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:19.775714 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:19.775149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:19.775714 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:19.775287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:19.775714 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:19.775407    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:19.775714 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:19.775511    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:20.941957 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.941745    2563 generic.go:356] "Generic (PLEG): container finished" podID="58aed374-015c-4ef8-aae2-2409c4fb847c" containerID="ead12777d1891a3325321e9b75d66382df01b0aa174dcb4da5e69d08b77b024a" exitCode=0
Sep 11 17:39:20.942568 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.941827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerDied","Data":"ead12777d1891a3325321e9b75d66382df01b0aa174dcb4da5e69d08b77b024a"}
Sep 11 17:39:20.945642 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.945622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:39:20.946046 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.946024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"c8b3f39f200ec4a467371b474b5955297a4a4de541c8ca094130077534385670"}
Sep 11 17:39:20.946298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.946272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:39:20.946472 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.946457    2563 scope.go:117] "RemoveContainer" containerID="d3ea30ab1a5ae3dfd56767da4cfab039d823de4e8ca0cd81333a69314b02f812"
Sep 11 17:39:20.962974 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:20.962953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:39:21.296362 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.296278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:39:21.776021 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.775834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:21.776124 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.775833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:21.776124 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:21.776108    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:21.776248 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.775859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:21.776327 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:21.776228    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:21.776393 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:21.776325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:21.910058 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.910017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:39:21.910232 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.910138    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 17:39:21.910677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.910651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wpcq7"
Sep 11 17:39:21.941568 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.941462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-94tdw"]
Sep 11 17:39:21.945629 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.945604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fxqxq"]
Sep 11 17:39:21.946251 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.946228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9shmh"]
Sep 11 17:39:21.952630 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.952607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:39:21.953065 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.953039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" event={"ID":"c1805cae-72e6-4e53-b586-3625d2c70393","Type":"ContainerStarted","Data":"54432891a38cda69c256a61f1d257c446f721f009f2b7b111bb93d7375494391"}
Sep 11 17:39:21.953390 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.953212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:39:21.955602 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.955578    2563 generic.go:356] "Generic (PLEG): container finished" podID="58aed374-015c-4ef8-aae2-2409c4fb847c" containerID="fe1a24c16cd86348e15002d863d7fd0f2571012b981311d46100d828efd06455" exitCode=0
Sep 11 17:39:21.955703 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.955657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:21.955703 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.955678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:21.955703 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.955668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerDied","Data":"fe1a24c16cd86348e15002d863d7fd0f2571012b981311d46100d828efd06455"}
Sep 11 17:39:21.955892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.955877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:21.956007 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:21.955986    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:21.956080 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:21.956056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:21.956148 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:21.956131    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:21.971181 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.971152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:39:21.988655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:21.986521    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98" podStartSLOduration=11.375140158 podStartE2EDuration="28.986450079s" totalImagesPullingTime="17.611309921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:21.986355916 +0000 UTC m=+28.749810647" watchObservedRunningTime="2026-09-11 17:39:21.986450079 +0000 UTC m=+28.749904807"
Sep 11 17:39:22.960540 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:22.960505    2563 generic.go:356] "Generic (PLEG): container finished" podID="58aed374-015c-4ef8-aae2-2409c4fb847c" containerID="b7e6643f040eeaaa5cec0c764922c928020b22ad240bccb04e312757f557aea4" exitCode=0
Sep 11 17:39:22.960958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:22.960592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerDied","Data":"b7e6643f040eeaaa5cec0c764922c928020b22ad240bccb04e312757f557aea4"}
Sep 11 17:39:23.776274 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:23.776243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:23.776546 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:23.776366    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:23.776775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:23.776753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:23.776909 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:23.776844    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:23.776909 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:23.776899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:23.777014 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:23.776958    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:24.131770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:24.131740    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kthkz_7f0836d0-c705-48f4-b420-cbada6b229a9/dns-node-resolver/0.log"
Sep 11 17:39:24.910802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:24.910775    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6mhjc_9c15f880-59bc-45ab-beff-5f7c3db77686/node-ca/0.log"
Sep 11 17:39:25.774991 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:25.774955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:25.775579 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:25.775091    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:25.775579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:25.774955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:25.775579 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:25.775181    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:25.775579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:25.774955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:25.775579 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:25.775253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:27.410355 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:27.410311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:27.410863 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:27.410371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:27.410863 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.410479    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:39:27.410863 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.410507    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:39:27.410863 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.410545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret podName:6e38817a-672b-4e85-ab7c-405f8eb555c5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:59.410531737 +0000 UTC m=+66.173986455 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret") pod "global-pull-secret-syncer-94tdw" (UID: "6e38817a-672b-4e85-ab7c-405f8eb555c5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:39:27.410863 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.410574    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs podName:cf89e371-caa1-4d99-b78e-d1e2110f4266 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:59.410556197 +0000 UTC m=+66.174010921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs") pod "network-metrics-daemon-fxqxq" (UID: "cf89e371-caa1-4d99-b78e-d1e2110f4266") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:39:27.611649 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:27.611614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:27.611835 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.611801    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:39:27.611835 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.611830    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:39:27.611962 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.611845    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5bjn9 for pod openshift-network-diagnostics/network-check-target-9shmh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:39:27.611962 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.611902    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9 podName:88395018-dee0-4289-97a0-d6313aa0ddc5 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:59.611885806 +0000 UTC m=+66.375340527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5bjn9" (UniqueName: "kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9") pod "network-check-target-9shmh" (UID: "88395018-dee0-4289-97a0-d6313aa0ddc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:39:27.775806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:27.775726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:27.775806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:27.775762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:27.776031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:27.775823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:27.776031 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.775944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:27.776121 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.776080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:27.776202 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:27.776182    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:28.976681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:28.976638    2563 generic.go:356] "Generic (PLEG): container finished" podID="58aed374-015c-4ef8-aae2-2409c4fb847c" containerID="f6ce9b3ae70b973eafb9d50aea6f3fe70288508dd3b7e9f19a3c0e6da9a7f84a" exitCode=0
Sep 11 17:39:28.976681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:28.976688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerDied","Data":"f6ce9b3ae70b973eafb9d50aea6f3fe70288508dd3b7e9f19a3c0e6da9a7f84a"}
Sep 11 17:39:29.775695 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:29.775657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:29.775900 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:29.775658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:29.775900 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:29.775797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:29.775900 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:29.775794    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:29.775900 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:29.775871    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:29.776125 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:29.775925    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:29.982106 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:29.982073    2563 generic.go:356] "Generic (PLEG): container finished" podID="58aed374-015c-4ef8-aae2-2409c4fb847c" containerID="3fc46a9ff775a5859177fffaadc9ddac80f4d9387a07e167f84b4f2f3f23ad31" exitCode=0
Sep 11 17:39:29.982466 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:29.982114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerDied","Data":"3fc46a9ff775a5859177fffaadc9ddac80f4d9387a07e167f84b4f2f3f23ad31"}
Sep 11 17:39:30.988111 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:30.988074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57r2c" event={"ID":"58aed374-015c-4ef8-aae2-2409c4fb847c","Type":"ContainerStarted","Data":"f68284f33b07316cd7a20bc86f6b21886745626693379b53b5fee892b4b6e00e"}
Sep 11 17:39:31.012149 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:31.012104    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-57r2c" podStartSLOduration=9.7641493 podStartE2EDuration="38.012086088s" totalImagesPullingTime="28.247936788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:31.010344746 +0000 UTC m=+37.773799475" watchObservedRunningTime="2026-09-11 17:39:31.012086088 +0000 UTC m=+37.775540815"
Sep 11 17:39:31.775574 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:31.775458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:31.775574 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:31.775484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:31.775574 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:31.775458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:31.775855 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:31.775607    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:31.775855 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:31.775743    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:31.775949 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:31.775856    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:33.775990 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:33.775955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:33.776453 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:33.776041    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:33.776453 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:33.776116    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:33.776453 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:33.776217    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:33.776453 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:33.776310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:33.776453 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:33.776413    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:35.775342 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:35.775304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:35.775810 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:35.775305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:35.775810 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:35.775422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:35.775810 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:35.775416    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:35.775810 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:35.775506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:35.775810 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:35.775605    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:37.775787 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:37.775749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:37.776277 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:37.775749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:37.776277 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:37.775871    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:37.776277 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:37.776000    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:37.776277 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:37.776014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:37.776277 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:37.776099    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:39.775834 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:39.775800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:39.776194 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:39.775812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:39.776194 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:39.775920    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:39.776194 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:39.775812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:39.776194 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:39.775991    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:39.776194 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:39.776056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:41.775137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:41.775102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:41.775627 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:41.775102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:41.775627 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:41.775214    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:41.775627 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:41.775107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:41.775627 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:41.775299    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:41.775627 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:41.775356    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:43.776345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:43.776309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:43.776815 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:43.776421    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:43.776815 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:43.776454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:43.776815 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:43.776572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:43.780449 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:43.776554    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:43.780449 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:43.778843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:45.775282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:45.775243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:45.775282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:45.775260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:45.775931 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:45.775260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:45.775931 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:45.775348    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94tdw" podUID="6e38817a-672b-4e85-ab7c-405f8eb555c5"
Sep 11 17:39:45.775931 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:45.775440    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fxqxq" podUID="cf89e371-caa1-4d99-b78e-d1e2110f4266"
Sep 11 17:39:45.775931 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:45.775516    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9shmh" podUID="88395018-dee0-4289-97a0-d6313aa0ddc5"
Sep 11 17:39:46.525218 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.525132    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-167.ec2.internal" event="NodeReady"
Sep 11 17:39:46.525369 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.525257    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:39:46.574684 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.574649    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-spwdq"]
Sep 11 17:39:46.597356 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.597255    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-578g9"]
Sep 11 17:39:46.597569 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.597362    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.604892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.604421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zb994\""
Sep 11 17:39:46.604892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.604503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:39:46.604892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.604505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:39:46.619205 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.619180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-spwdq"]
Sep 11 17:39:46.619334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.619216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-578g9"]
Sep 11 17:39:46.619334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.619287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.620442 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.620420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xpfvj"]
Sep 11 17:39:46.622739 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.622711    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:39:46.623094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.623082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-72jzj\""
Sep 11 17:39:46.623361 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.623348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:39:46.623361 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.623357    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:39:46.623507 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.623387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:39:46.639527 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.639476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.644076 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.644053    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:39:46.644516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.644191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:39:46.644516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.644303    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-66cbf\""
Sep 11 17:39:46.644516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.644396    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:39:46.644516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.644420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpfvj"]
Sep 11 17:39:46.745006 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.744964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-config-volume\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.745006 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb26w\" (UniqueName: \"kubernetes.io/projected/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-kube-api-access-fb26w\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.745262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-data-volume\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.745262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bgb\" (UniqueName: \"kubernetes.io/projected/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-kube-api-access-q8bgb\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.745262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-crio-socket\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.745262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.745262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-tmp-dir\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.745262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-metrics-tls\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.745467 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.745467 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447fff49-0885-43e0-9b38-df3d55ad4a92-cert\") pod \"ingress-canary-xpfvj\" (UID: \"447fff49-0885-43e0-9b38-df3d55ad4a92\") " pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.745467 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.745323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtwp\" (UniqueName: \"kubernetes.io/projected/447fff49-0885-43e0-9b38-df3d55ad4a92-kube-api-access-vrtwp\") pod \"ingress-canary-xpfvj\" (UID: \"447fff49-0885-43e0-9b38-df3d55ad4a92\") " pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.846365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-data-volume\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bgb\" (UniqueName: \"kubernetes.io/projected/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-kube-api-access-q8bgb\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-crio-socket\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-tmp-dir\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-metrics-tls\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-crio-socket\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.846802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447fff49-0885-43e0-9b38-df3d55ad4a92-cert\") pod \"ingress-canary-xpfvj\" (UID: \"447fff49-0885-43e0-9b38-df3d55ad4a92\") " pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.847188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtwp\" (UniqueName: \"kubernetes.io/projected/447fff49-0885-43e0-9b38-df3d55ad4a92-kube-api-access-vrtwp\") pod \"ingress-canary-xpfvj\" (UID: \"447fff49-0885-43e0-9b38-df3d55ad4a92\") " pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.847188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-config-volume\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.847188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb26w\" (UniqueName: \"kubernetes.io/projected/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-kube-api-access-fb26w\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.847188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.846974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-tmp-dir\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.847314 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.847198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.847387 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.847368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-config-volume\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.850744 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.850722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447fff49-0885-43e0-9b38-df3d55ad4a92-cert\") pod \"ingress-canary-xpfvj\" (UID: \"447fff49-0885-43e0-9b38-df3d55ad4a92\") " pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.850851 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.850742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.850851 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.850742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-metrics-tls\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.856329 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.856306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bgb\" (UniqueName: \"kubernetes.io/projected/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-kube-api-access-q8bgb\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.856433 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.856411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb26w\" (UniqueName: \"kubernetes.io/projected/9c11678b-c630-4a52-b7b2-f4bb6c9f98bd-kube-api-access-fb26w\") pod \"dns-default-spwdq\" (UID: \"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd\") " pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.856516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.856482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtwp\" (UniqueName: \"kubernetes.io/projected/447fff49-0885-43e0-9b38-df3d55ad4a92-kube-api-access-vrtwp\") pod \"ingress-canary-xpfvj\" (UID: \"447fff49-0885-43e0-9b38-df3d55ad4a92\") " pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:46.863794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.863776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c530f3de-de46-4ba9-80de-5bbe9bdc6b62-data-volume\") pod \"insights-runtime-extractor-578g9\" (UID: \"c530f3de-de46-4ba9-80de-5bbe9bdc6b62\") " pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.907127 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.907092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:46.927626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.927597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-578g9"
Sep 11 17:39:46.948335 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:46.948306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpfvj"
Sep 11 17:39:47.246396 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.246317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-578g9"]
Sep 11 17:39:47.252942 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:47.252917    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc530f3de_de46_4ba9_80de_5bbe9bdc6b62.slice/crio-408dae512c3d38b6d0a953d0cfe1121847597049472e06f0233b23265764c342 WatchSource:0}: Error finding container 408dae512c3d38b6d0a953d0cfe1121847597049472e06f0233b23265764c342: Status 404 returned error can't find the container with id 408dae512c3d38b6d0a953d0cfe1121847597049472e06f0233b23265764c342
Sep 11 17:39:47.255029 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.255008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-spwdq"]
Sep 11 17:39:47.257794 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:47.257774    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c11678b_c630_4a52_b7b2_f4bb6c9f98bd.slice/crio-67239ebce1ee6dd14014409f940917582d59405174c0c71930b9aad63f14d88b WatchSource:0}: Error finding container 67239ebce1ee6dd14014409f940917582d59405174c0c71930b9aad63f14d88b: Status 404 returned error can't find the container with id 67239ebce1ee6dd14014409f940917582d59405174c0c71930b9aad63f14d88b
Sep 11 17:39:47.279338 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.279314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpfvj"]
Sep 11 17:39:47.282578 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:47.282554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447fff49_0885_43e0_9b38_df3d55ad4a92.slice/crio-cbce3ee6510909d61e937d028c62bb66c3bb7d19b6c12dca2a54b61d4276fa1a WatchSource:0}: Error finding container cbce3ee6510909d61e937d028c62bb66c3bb7d19b6c12dca2a54b61d4276fa1a: Status 404 returned error can't find the container with id cbce3ee6510909d61e937d028c62bb66c3bb7d19b6c12dca2a54b61d4276fa1a
Sep 11 17:39:47.775452 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.775414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:47.775742 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.775414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:47.775806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.775414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:47.779725 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.779684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:39:47.779894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.779793    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:39:47.779894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.779797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hwqhm\""
Sep 11 17:39:47.779894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.779809    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:39:47.779894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.779809    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:39:47.780139 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:47.779948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7q2fl\""
Sep 11 17:39:48.031443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:48.031363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578g9" event={"ID":"c530f3de-de46-4ba9-80de-5bbe9bdc6b62","Type":"ContainerStarted","Data":"89471068345fd99770d8f8e5e1499e86f9e0838f7726c417758464545ef210f5"}
Sep 11 17:39:48.031443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:48.031432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578g9" event={"ID":"c530f3de-de46-4ba9-80de-5bbe9bdc6b62","Type":"ContainerStarted","Data":"408dae512c3d38b6d0a953d0cfe1121847597049472e06f0233b23265764c342"}
Sep 11 17:39:48.034045 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:48.033969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spwdq" event={"ID":"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd","Type":"ContainerStarted","Data":"67239ebce1ee6dd14014409f940917582d59405174c0c71930b9aad63f14d88b"}
Sep 11 17:39:48.036705 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:48.036360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpfvj" event={"ID":"447fff49-0885-43e0-9b38-df3d55ad4a92","Type":"ContainerStarted","Data":"cbce3ee6510909d61e937d028c62bb66c3bb7d19b6c12dca2a54b61d4276fa1a"}
Sep 11 17:39:49.041316 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:49.041282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578g9" event={"ID":"c530f3de-de46-4ba9-80de-5bbe9bdc6b62","Type":"ContainerStarted","Data":"6469dbe082fe25a3c91c0cc25c69964150e5203b20a6cdd9b42ec1b1796a339f"}
Sep 11 17:39:50.052015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.050508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spwdq" event={"ID":"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd","Type":"ContainerStarted","Data":"ed807316c69181c8e443ae45a34faddfdb8789e1d2842c569948c86d169f7b5a"}
Sep 11 17:39:50.052015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.050581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-spwdq" event={"ID":"9c11678b-c630-4a52-b7b2-f4bb6c9f98bd","Type":"ContainerStarted","Data":"6169f0ff3ddca070c6252eca4a9e1c0575077fd3283148d2426f68256812bc38"}
Sep 11 17:39:50.052015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.051465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-spwdq"
Sep 11 17:39:50.054322 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.054291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpfvj" event={"ID":"447fff49-0885-43e0-9b38-df3d55ad4a92","Type":"ContainerStarted","Data":"592e14ced770c56bc9b549da1388bdf99949d81952f80721c956027f314c144a"}
Sep 11 17:39:50.076591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.076515    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-spwdq" podStartSLOduration=1.545581799 podStartE2EDuration="4.076483445s" totalImagesPullingTime="2.530901646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:50.074578678 +0000 UTC m=+56.838033406" watchObservedRunningTime="2026-09-11 17:39:50.076483445 +0000 UTC m=+56.839938171"
Sep 11 17:39:50.095415 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.095360    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xpfvj" podStartSLOduration=1.54579172 podStartE2EDuration="4.09534182s" totalImagesPullingTime="2.5495501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:50.094076546 +0000 UTC m=+56.857531275" watchObservedRunningTime="2026-09-11 17:39:50.09534182 +0000 UTC m=+56.858796538"
Sep 11 17:39:50.319755 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.319725    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"]
Sep 11 17:39:50.322828 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.322804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:50.333698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.333676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zp4vw\""
Sep 11 17:39:50.334170 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.334150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 17:39:50.336217 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.336191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"]
Sep 11 17:39:50.475521 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.475466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/226aa17f-6264-4f54-a35a-861ccc9bd9be-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-kvcsq\" (UID: \"226aa17f-6264-4f54-a35a-861ccc9bd9be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:50.576250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.576172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/226aa17f-6264-4f54-a35a-861ccc9bd9be-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-kvcsq\" (UID: \"226aa17f-6264-4f54-a35a-861ccc9bd9be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:50.578999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.578973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/226aa17f-6264-4f54-a35a-861ccc9bd9be-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-kvcsq\" (UID: \"226aa17f-6264-4f54-a35a-861ccc9bd9be\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:50.632892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:50.632861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:51.005371 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:51.005329    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"]
Sep 11 17:39:51.007963 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:51.007931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226aa17f_6264_4f54_a35a_861ccc9bd9be.slice/crio-4650719eeb3c8d3d6e8bc95443abfc92c54792de4fdaf63983817d2767833f65 WatchSource:0}: Error finding container 4650719eeb3c8d3d6e8bc95443abfc92c54792de4fdaf63983817d2767833f65: Status 404 returned error can't find the container with id 4650719eeb3c8d3d6e8bc95443abfc92c54792de4fdaf63983817d2767833f65
Sep 11 17:39:51.060128 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:51.060087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-578g9" event={"ID":"c530f3de-de46-4ba9-80de-5bbe9bdc6b62","Type":"ContainerStarted","Data":"7638e258e3322484e414e861a3de9631650e0243bfbd721a1bf8ab62e97e02c7"}
Sep 11 17:39:51.061414 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:51.061393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq" event={"ID":"226aa17f-6264-4f54-a35a-861ccc9bd9be","Type":"ContainerStarted","Data":"4650719eeb3c8d3d6e8bc95443abfc92c54792de4fdaf63983817d2767833f65"}
Sep 11 17:39:51.082562 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:51.082512    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-578g9" podStartSLOduration=1.7546140289999999 podStartE2EDuration="5.082475854s" totalImagesPullingTime="3.327861825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:39:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:51.081027642 +0000 UTC m=+57.844482368" watchObservedRunningTime="2026-09-11 17:39:51.082475854 +0000 UTC m=+57.845930580"
Sep 11 17:39:53.071796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.071758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq" event={"ID":"226aa17f-6264-4f54-a35a-861ccc9bd9be","Type":"ContainerStarted","Data":"d5990ca171169fe1862d9c74d6e41ca34f0ce6cad1a424ca0cf6bba1bee9a2d0"}
Sep 11 17:39:53.072183 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.071973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:53.076665 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.076640    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq"
Sep 11 17:39:53.094922 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.094863    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-kvcsq" podStartSLOduration=1.967066933 podStartE2EDuration="3.094846686s" totalImagesPullingTime="1.127779753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:53.094529252 +0000 UTC m=+59.857983980" watchObservedRunningTime="2026-09-11 17:39:53.094846686 +0000 UTC m=+59.858301413"
Sep 11 17:39:53.439631 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.439551    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"]
Sep 11 17:39:53.442582 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.442567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.448578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.448548    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:39:53.448661 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.448575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:39:53.448661 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.448630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:39:53.448758 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.448580    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-868xg\""
Sep 11 17:39:53.448913 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.448805    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:39:53.448989 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.448972    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:39:53.473743 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.473715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"]
Sep 11 17:39:53.596020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.595974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bea627ed-cd42-4058-9be4-f51e58deea6b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.596020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.596023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzk7w\" (UniqueName: \"kubernetes.io/projected/bea627ed-cd42-4058-9be4-f51e58deea6b-kube-api-access-qzk7w\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.596296 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.596185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.596296 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.596223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.696988 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.696898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.696988 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.696940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.696988 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.696973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bea627ed-cd42-4058-9be4-f51e58deea6b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.697252 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.696995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzk7w\" (UniqueName: \"kubernetes.io/projected/bea627ed-cd42-4058-9be4-f51e58deea6b-kube-api-access-qzk7w\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.700104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.700084    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:39:53.700194 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.700148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:39:53.700194 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.700174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:39:53.707179 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:53.707160    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 17:39:53.707301 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:53.707250    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-tls podName:bea627ed-cd42-4058-9be4-f51e58deea6b nodeName:}" failed. No retries permitted until 2026-09-11 17:39:54.207228926 +0000 UTC m=+60.970683648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-wcflv" (UID: "bea627ed-cd42-4058-9be4-f51e58deea6b") : secret "prometheus-operator-tls" not found
Sep 11 17:39:53.707964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.707946    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:39:53.708506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.708470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bea627ed-cd42-4058-9be4-f51e58deea6b-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.709745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.709720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.717664 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.717635    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:39:53.728146 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.728117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzk7w\" (UniqueName: \"kubernetes.io/projected/bea627ed-cd42-4058-9be4-f51e58deea6b-kube-api-access-qzk7w\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:53.974304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:53.974225    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9zj98"
Sep 11 17:39:54.301333 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:54.301236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:54.303727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:54.303701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bea627ed-cd42-4058-9be4-f51e58deea6b-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-wcflv\" (UID: \"bea627ed-cd42-4058-9be4-f51e58deea6b\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:54.353892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:54.353863    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-868xg\""
Sep 11 17:39:54.361502 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:54.361467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"
Sep 11 17:39:54.634087 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:54.634057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-wcflv"]
Sep 11 17:39:54.636959 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:54.636931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea627ed_cd42_4058_9be4_f51e58deea6b.slice/crio-5b0c9af57521c52792082a0c6fbcaf65efb50b47ad9e78323e27266e0125337d WatchSource:0}: Error finding container 5b0c9af57521c52792082a0c6fbcaf65efb50b47ad9e78323e27266e0125337d: Status 404 returned error can't find the container with id 5b0c9af57521c52792082a0c6fbcaf65efb50b47ad9e78323e27266e0125337d
Sep 11 17:39:55.080079 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:55.080036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv" event={"ID":"bea627ed-cd42-4058-9be4-f51e58deea6b","Type":"ContainerStarted","Data":"5b0c9af57521c52792082a0c6fbcaf65efb50b47ad9e78323e27266e0125337d"}
Sep 11 17:39:57.088740 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:57.088700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv" event={"ID":"bea627ed-cd42-4058-9be4-f51e58deea6b","Type":"ContainerStarted","Data":"995406a8ff4ae6451e68dda7ca10178847c1f423512833a2d7a595a5d0b0941e"}
Sep 11 17:39:57.089134 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:57.088800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv" event={"ID":"bea627ed-cd42-4058-9be4-f51e58deea6b","Type":"ContainerStarted","Data":"0fb66d6d78050d6d5d016da997f6e12d862199ee30c6104948a801f6b71258ce"}
Sep 11 17:39:57.108182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:57.108130    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-wcflv" podStartSLOduration=2.60144944 podStartE2EDuration="4.108114411s" totalImagesPullingTime="1.506664971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:57.106635227 +0000 UTC m=+63.870089967" watchObservedRunningTime="2026-09-11 17:39:57.108114411 +0000 UTC m=+63.871569137"
Sep 11 17:39:58.822648 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.822609    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"]
Sep 11 17:39:58.826023 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.826002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.830542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.830515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sldkl\""
Sep 11 17:39:58.830688 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.830544    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 17:39:58.830688 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.830564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 17:39:58.830688 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.830580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 17:39:58.833133 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.833110    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jc6m6"]
Sep 11 17:39:58.836511 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.836479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.839306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.839287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:39:58.839395 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.839342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:39:58.839617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.839603    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m696x\""
Sep 11 17:39:58.839886 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.839872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:39:58.847177 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.847157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"]
Sep 11 17:39:58.935340 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-textfile\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5j8\" (UniqueName: \"kubernetes.io/projected/9e3d2659-dc80-4b14-a6bc-c7292398abc9-kube-api-access-ft5j8\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c1f41e-0109-4f2f-b5e1-d26237330394-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.935523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29c1f41e-0109-4f2f-b5e1-d26237330394-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.935523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-wtmp\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.935657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wzd\" (UniqueName: \"kubernetes.io/projected/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-api-access-24wzd\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.935657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-root\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.935657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:58.935657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-tls\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935858 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e3d2659-dc80-4b14-a6bc-c7292398abc9-metrics-client-ca\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935858 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-sys\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935858 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:58.935858 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:58.935795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-accelerators-collector-config\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037019 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.036982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29c1f41e-0109-4f2f-b5e1-d26237330394-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.037019 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-wtmp\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037019 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.037282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-wtmp\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wzd\" (UniqueName: \"kubernetes.io/projected/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-api-access-24wzd\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.037282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-root\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.037282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.037282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-root\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-tls\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e3d2659-dc80-4b14-a6bc-c7292398abc9-metrics-client-ca\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:59.037393    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-sys\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29c1f41e-0109-4f2f-b5e1-d26237330394-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e3d2659-dc80-4b14-a6bc-c7292398abc9-sys\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:39:59.037552    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-tls podName:29c1f41e-0109-4f2f-b5e1-d26237330394 nodeName:}" failed. No retries permitted until 2026-09-11 17:39:59.537530255 +0000 UTC m=+66.300984965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-r9lqm" (UID: "29c1f41e-0109-4f2f-b5e1-d26237330394") : secret "kube-state-metrics-tls" not found
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-accelerators-collector-config\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.037606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-textfile\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.038098 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5j8\" (UniqueName: \"kubernetes.io/projected/9e3d2659-dc80-4b14-a6bc-c7292398abc9-kube-api-access-ft5j8\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.038098 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c1f41e-0109-4f2f-b5e1-d26237330394-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.038098 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.038098 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.037993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-textfile\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.038469 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.038104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e3d2659-dc80-4b14-a6bc-c7292398abc9-metrics-client-ca\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.038469 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.038136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-accelerators-collector-config\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.038469 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.038263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c1f41e-0109-4f2f-b5e1-d26237330394-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.040069 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.040043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-tls\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.040346 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.040329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.040411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.040343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e3d2659-dc80-4b14-a6bc-c7292398abc9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.047168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.047141    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5j8\" (UniqueName: \"kubernetes.io/projected/9e3d2659-dc80-4b14-a6bc-c7292398abc9-kube-api-access-ft5j8\") pod \"node-exporter-jc6m6\" (UID: \"9e3d2659-dc80-4b14-a6bc-c7292398abc9\") " pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.047296 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.047278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wzd\" (UniqueName: \"kubernetes.io/projected/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-api-access-24wzd\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.144251 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.144164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jc6m6"
Sep 11 17:39:59.156526 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:59.156478    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3d2659_dc80_4b14_a6bc_c7292398abc9.slice/crio-d73f6904fb92d8d14ca7d8da29b831c61062e70ce74cdd39acc0640e814e32c0 WatchSource:0}: Error finding container d73f6904fb92d8d14ca7d8da29b831c61062e70ce74cdd39acc0640e814e32c0: Status 404 returned error can't find the container with id d73f6904fb92d8d14ca7d8da29b831c61062e70ce74cdd39acc0640e814e32c0
Sep 11 17:39:59.440324 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.440228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:59.440324 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.440303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:59.443228 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.443207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:39:59.443290 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.443237    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:39:59.453360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.453334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e38817a-672b-4e85-ab7c-405f8eb555c5-original-pull-secret\") pod \"global-pull-secret-syncer-94tdw\" (UID: \"6e38817a-672b-4e85-ab7c-405f8eb555c5\") " pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:59.453470 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.453366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf89e371-caa1-4d99-b78e-d1e2110f4266-metrics-certs\") pod \"network-metrics-daemon-fxqxq\" (UID: \"cf89e371-caa1-4d99-b78e-d1e2110f4266\") " pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:59.493152 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.493112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94tdw"
Sep 11 17:39:59.501006 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.500980    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7q2fl\""
Sep 11 17:39:59.509321 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.509296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fxqxq"
Sep 11 17:39:59.541296 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.541265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.543821 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.543799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29c1f41e-0109-4f2f-b5e1-d26237330394-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-r9lqm\" (UID: \"29c1f41e-0109-4f2f-b5e1-d26237330394\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.642539 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.642468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:59.645604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.645586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:39:59.655955 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.655929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:39:59.666234 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.666203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjn9\" (UniqueName: \"kubernetes.io/projected/88395018-dee0-4289-97a0-d6313aa0ddc5-kube-api-access-5bjn9\") pod \"network-check-target-9shmh\" (UID: \"88395018-dee0-4289-97a0-d6313aa0ddc5\") " pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:59.734827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.734747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"
Sep 11 17:39:59.790257 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.790059    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hwqhm\""
Sep 11 17:39:59.795687 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.795657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-94tdw"]
Sep 11 17:39:59.796794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.796762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:39:59.800031 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:59.799996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e38817a_672b_4e85_ab7c_405f8eb555c5.slice/crio-7001f2d31352ae91f2028afb34a2113c7b832a9f8779b797e0dcb40f5ceb26c8 WatchSource:0}: Error finding container 7001f2d31352ae91f2028afb34a2113c7b832a9f8779b797e0dcb40f5ceb26c8: Status 404 returned error can't find the container with id 7001f2d31352ae91f2028afb34a2113c7b832a9f8779b797e0dcb40f5ceb26c8
Sep 11 17:39:59.820284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.820197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fxqxq"]
Sep 11 17:39:59.825213 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:39:59.825132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf89e371_caa1_4d99_b78e_d1e2110f4266.slice/crio-1216550a054dbd40262a9c531ab394684b1a9f778ed227bf674e8eb81c5c3b8a WatchSource:0}: Error finding container 1216550a054dbd40262a9c531ab394684b1a9f778ed227bf674e8eb81c5c3b8a: Status 404 returned error can't find the container with id 1216550a054dbd40262a9c531ab394684b1a9f778ed227bf674e8eb81c5c3b8a
Sep 11 17:39:59.918640 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.918609    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:39:59.941482 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.941436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:39:59.948829 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.948800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:39:59.949657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.949096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:39:59.949657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.949316    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:39:59.949657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.949538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jg7mx\""
Sep 11 17:39:59.949657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.949534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:39:59.949941 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.949737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:39:59.950069 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.950047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:39:59.950139 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.950077    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:39:59.950139 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.950121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:39:59.950443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.950423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:39:59.950581 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:39:59.950520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:40:00.049691 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.049691 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.049895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.049895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.049895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.049895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4g7\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-kube-api-access-jb4g7\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-web-config\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.049951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.050003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.050045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-out\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.050084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.050298 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.050115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.053288 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.053255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm"]
Sep 11 17:40:00.057205 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:00.057178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c1f41e_0109_4f2f_b5e1_d26237330394.slice/crio-a53862d42295e31e67aa1c82119b18c04231bf96d9f2cab74f74825aa98ce079 WatchSource:0}: Error finding container a53862d42295e31e67aa1c82119b18c04231bf96d9f2cab74f74825aa98ce079: Status 404 returned error can't find the container with id a53862d42295e31e67aa1c82119b18c04231bf96d9f2cab74f74825aa98ce079
Sep 11 17:40:00.100584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.100551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-94tdw" event={"ID":"6e38817a-672b-4e85-ab7c-405f8eb555c5","Type":"ContainerStarted","Data":"7001f2d31352ae91f2028afb34a2113c7b832a9f8779b797e0dcb40f5ceb26c8"}
Sep 11 17:40:00.101782 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.101753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jc6m6" event={"ID":"9e3d2659-dc80-4b14-a6bc-c7292398abc9","Type":"ContainerStarted","Data":"d73f6904fb92d8d14ca7d8da29b831c61062e70ce74cdd39acc0640e814e32c0"}
Sep 11 17:40:00.102944 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.102923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm" event={"ID":"29c1f41e-0109-4f2f-b5e1-d26237330394","Type":"ContainerStarted","Data":"a53862d42295e31e67aa1c82119b18c04231bf96d9f2cab74f74825aa98ce079"}
Sep 11 17:40:00.104662 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.104626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fxqxq" event={"ID":"cf89e371-caa1-4d99-b78e-d1e2110f4266","Type":"ContainerStarted","Data":"1216550a054dbd40262a9c531ab394684b1a9f778ed227bf674e8eb81c5c3b8a"}
Sep 11 17:40:00.105770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.105747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9shmh"]
Sep 11 17:40:00.151293 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-out\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151623 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151623 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151623 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151623 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4g7\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-kube-api-access-jb4g7\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-web-config\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.151827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.152124 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.151898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.153525 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.153441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.153709 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.153618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.154741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.154695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.155715 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.155689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.155837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.155816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.156122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.156065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-out\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.156838 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.156483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.156838 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.156783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.156999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.156847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.157055 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.157041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-web-config\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.158120 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.158098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.161356 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.161336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4g7\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-kube-api-access-jb4g7\") pod \"alertmanager-main-0\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.258077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.258037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:40:00.397810 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:00.397767    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88395018_dee0_4289_97a0_d6313aa0ddc5.slice/crio-103b2aa72b7ffbddd9208cdf3dbeb9580a20f64ce49c32b4b953855239d31fde WatchSource:0}: Error finding container 103b2aa72b7ffbddd9208cdf3dbeb9580a20f64ce49c32b4b953855239d31fde: Status 404 returned error can't find the container with id 103b2aa72b7ffbddd9208cdf3dbeb9580a20f64ce49c32b4b953855239d31fde
Sep 11 17:40:00.680592 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:00.680506    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:40:00.685207 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:00.685167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0e684e_f6b7_403b_8ca8_f7e67afd1c00.slice/crio-487ac353bc3b51680b2aa70fa884a9d8b56d4456004162a0b9337b5b654c266e WatchSource:0}: Error finding container 487ac353bc3b51680b2aa70fa884a9d8b56d4456004162a0b9337b5b654c266e: Status 404 returned error can't find the container with id 487ac353bc3b51680b2aa70fa884a9d8b56d4456004162a0b9337b5b654c266e
Sep 11 17:40:01.067483 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:01.067410    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-spwdq"
Sep 11 17:40:01.111366 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:01.111329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"487ac353bc3b51680b2aa70fa884a9d8b56d4456004162a0b9337b5b654c266e"}
Sep 11 17:40:01.117254 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:01.116582    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e3d2659-dc80-4b14-a6bc-c7292398abc9" containerID="665523c53ae8331faf716415ac4bcbf75276bb4deab1c21709074b07594b19fa" exitCode=0
Sep 11 17:40:01.117254 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:01.116850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jc6m6" event={"ID":"9e3d2659-dc80-4b14-a6bc-c7292398abc9","Type":"ContainerDied","Data":"665523c53ae8331faf716415ac4bcbf75276bb4deab1c21709074b07594b19fa"}
Sep 11 17:40:01.119071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:01.119046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9shmh" event={"ID":"88395018-dee0-4289-97a0-d6313aa0ddc5","Type":"ContainerStarted","Data":"103b2aa72b7ffbddd9208cdf3dbeb9580a20f64ce49c32b4b953855239d31fde"}
Sep 11 17:40:02.128555 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.128518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fxqxq" event={"ID":"cf89e371-caa1-4d99-b78e-d1e2110f4266","Type":"ContainerStarted","Data":"a5ba1168bf530a85d26496c1e1566b1b3e64edf9a4d12f0829eff97616eac014"}
Sep 11 17:40:02.129065 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.128600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fxqxq" event={"ID":"cf89e371-caa1-4d99-b78e-d1e2110f4266","Type":"ContainerStarted","Data":"266df1208a15499d68dda115e8ddafb27aea0d0e50fb813be1ba2df5b63c886a"}
Sep 11 17:40:02.132984 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.132952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jc6m6" event={"ID":"9e3d2659-dc80-4b14-a6bc-c7292398abc9","Type":"ContainerStarted","Data":"f1a22723a007fe8a48b602af8865c7872d5c38cab359c3c7a83ba285ce5cb39a"}
Sep 11 17:40:02.133159 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.133016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jc6m6" event={"ID":"9e3d2659-dc80-4b14-a6bc-c7292398abc9","Type":"ContainerStarted","Data":"ff95c6da1cb65121b0348aa030130fe08b1056b977da50cd19fe8af7da18861e"}
Sep 11 17:40:02.136737 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.136711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm" event={"ID":"29c1f41e-0109-4f2f-b5e1-d26237330394","Type":"ContainerStarted","Data":"1f4daaee56822fec0d2dd4070ea7c1dd60e2bf14f01b7e437fece73d2d0380bc"}
Sep 11 17:40:02.136865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.136760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm" event={"ID":"29c1f41e-0109-4f2f-b5e1-d26237330394","Type":"ContainerStarted","Data":"15a8ebab755cc039ebff606902ac53fe0c5f914488806f0ff4cede8f15702688"}
Sep 11 17:40:02.136865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.136780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm" event={"ID":"29c1f41e-0109-4f2f-b5e1-d26237330394","Type":"ContainerStarted","Data":"a3b3b1f546937b08a02d98b72390bc63a2c5795bf733c48c3cc6b996eecde305"}
Sep 11 17:40:02.196043 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.195964    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fxqxq" podStartSLOduration=67.696282796 podStartE2EDuration="1m9.195937739s" totalImagesPullingTime="1.499654943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:02.158638962 +0000 UTC m=+68.922093691" watchObservedRunningTime="2026-09-11 17:40:02.195937739 +0000 UTC m=+68.959392465"
Sep 11 17:40:02.229756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.229690    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jc6m6" podStartSLOduration=2.6105002859999997 podStartE2EDuration="4.22966843s" totalImagesPullingTime="1.531575461s" totalInitContainerRuntime="87.592683ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:02.229626967 +0000 UTC m=+68.993081694" watchObservedRunningTime="2026-09-11 17:40:02.22966843 +0000 UTC m=+68.993123158"
Sep 11 17:40:02.230086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:02.230047    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-r9lqm" podStartSLOduration=2.451522952 podStartE2EDuration="4.230034363s" totalImagesPullingTime="1.778511411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:02.197218977 +0000 UTC m=+68.960673704" watchObservedRunningTime="2026-09-11 17:40:02.230034363 +0000 UTC m=+68.993489091"
Sep 11 17:40:03.143706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.143665    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5" exitCode=0
Sep 11 17:40:03.144161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.143770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"}
Sep 11 17:40:03.432009 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.430790    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-645b98cdc6-8g2vt"]
Sep 11 17:40:03.434506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.434469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.440272 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.440238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 17:40:03.440549 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.440531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zk2kf\""
Sep 11 17:40:03.440709 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.440695    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 17:40:03.440866 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.440849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 17:40:03.441137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.441122    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fql7lbsvo01fb\""
Sep 11 17:40:03.441411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.441394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 17:40:03.446786 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.446762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-645b98cdc6-8g2vt"]
Sep 11 17:40:03.481638 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d952eb63-5524-4d2b-b9c5-c40ce20966ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.481827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d952eb63-5524-4d2b-b9c5-c40ce20966ba-audit-log\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.481827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn7dr\" (UniqueName: \"kubernetes.io/projected/d952eb63-5524-4d2b-b9c5-c40ce20966ba-kube-api-access-sn7dr\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.481827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-client-ca-bundle\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.481827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-secret-metrics-server-client-certs\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.481827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-secret-metrics-server-tls\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.481827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.481802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d952eb63-5524-4d2b-b9c5-c40ce20966ba-metrics-server-audit-profiles\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.582867 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.582831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-client-ca-bundle\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583051 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.582889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-secret-metrics-server-client-certs\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583051 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.582925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-secret-metrics-server-tls\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583051 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.582955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d952eb63-5524-4d2b-b9c5-c40ce20966ba-metrics-server-audit-profiles\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583051 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.582989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d952eb63-5524-4d2b-b9c5-c40ce20966ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583051 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.583020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d952eb63-5524-4d2b-b9c5-c40ce20966ba-audit-log\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.583058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn7dr\" (UniqueName: \"kubernetes.io/projected/d952eb63-5524-4d2b-b9c5-c40ce20966ba-kube-api-access-sn7dr\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.583306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.583133    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"]
Sep 11 17:40:03.585265 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.584693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d952eb63-5524-4d2b-b9c5-c40ce20966ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.585265 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.584973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d952eb63-5524-4d2b-b9c5-c40ce20966ba-audit-log\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.585265 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.585030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d952eb63-5524-4d2b-b9c5-c40ce20966ba-metrics-server-audit-profiles\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.586725 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.586703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.588218 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.588192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-secret-metrics-server-client-certs\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.588891 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.588817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-secret-metrics-server-tls\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.589509 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.589456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d952eb63-5524-4d2b-b9c5-c40ce20966ba-client-ca-bundle\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.589639 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.589621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 17:40:03.589719 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.589682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mbr74\""
Sep 11 17:40:03.595851 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.595823    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"]
Sep 11 17:40:03.599279 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.599233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn7dr\" (UniqueName: \"kubernetes.io/projected/d952eb63-5524-4d2b-b9c5-c40ce20966ba-kube-api-access-sn7dr\") pod \"metrics-server-645b98cdc6-8g2vt\" (UID: \"d952eb63-5524-4d2b-b9c5-c40ce20966ba\") " pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.683512 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.683402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xptdx\" (UniqueName: \"kubernetes.io/projected/7c1f8b56-129e-4aac-ba8f-6933c3b84139-kube-api-access-xptdx\") pod \"monitoring-plugin-5d85778b9d-q7pnl\" (UID: \"7c1f8b56-129e-4aac-ba8f-6933c3b84139\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.683512 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.683443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c1f8b56-129e-4aac-ba8f-6933c3b84139-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-q7pnl\" (UID: \"7c1f8b56-129e-4aac-ba8f-6933c3b84139\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.749612 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.749574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:03.784382 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.784331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xptdx\" (UniqueName: \"kubernetes.io/projected/7c1f8b56-129e-4aac-ba8f-6933c3b84139-kube-api-access-xptdx\") pod \"monitoring-plugin-5d85778b9d-q7pnl\" (UID: \"7c1f8b56-129e-4aac-ba8f-6933c3b84139\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.784382 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.784378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c1f8b56-129e-4aac-ba8f-6933c3b84139-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-q7pnl\" (UID: \"7c1f8b56-129e-4aac-ba8f-6933c3b84139\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.787321 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.787289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c1f8b56-129e-4aac-ba8f-6933c3b84139-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-q7pnl\" (UID: \"7c1f8b56-129e-4aac-ba8f-6933c3b84139\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.795703 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.795672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xptdx\" (UniqueName: \"kubernetes.io/projected/7c1f8b56-129e-4aac-ba8f-6933c3b84139-kube-api-access-xptdx\") pod \"monitoring-plugin-5d85778b9d-q7pnl\" (UID: \"7c1f8b56-129e-4aac-ba8f-6933c3b84139\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.901088 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.901048    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75cc6f7574-9mjmk"]
Sep 11 17:40:03.904233 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.904201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.907595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.907567    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 17:40:03.907792 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.907763    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-569cd\""
Sep 11 17:40:03.907907 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.907892    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:40:03.908209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.908139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:40:03.908209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.908179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:40:03.908209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.908197    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:40:03.908595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.908214    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:40:03.908595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.908142    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 17:40:03.914197 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.914156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:40:03.915473 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.915450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75cc6f7574-9mjmk"]
Sep 11 17:40:03.923065 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.923031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:03.986242 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-service-ca\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.986242 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-oauth-serving-cert\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.986242 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdw9d\" (UniqueName: \"kubernetes.io/projected/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-kube-api-access-rdw9d\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.986538 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986322    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-config\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.986538 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-trusted-ca-bundle\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.986538 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-oauth-config\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:03.986538 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:03.986442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-serving-cert\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.022603 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.022568    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"]
Sep 11 17:40:04.028323 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.028289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.031764 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.031738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 17:40:04.031909 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.031744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 17:40:04.032839 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.032750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 17:40:04.032969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.032952    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 17:40:04.033570 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.033547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n4qdk\""
Sep 11 17:40:04.034029 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.034008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 17:40:04.039302 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.039284    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 17:40:04.043607 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.043581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"]
Sep 11 17:40:04.087892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.087862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dtf\" (UniqueName: \"kubernetes.io/projected/c19e5383-0743-4d11-9b1a-fceb9969556a-kube-api-access-k8dtf\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.087903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-metrics-client-ca\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.087945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-serving-cert\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.087986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-telemeter-client-tls\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-service-ca\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-federate-client-tls\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-secret-telemeter-client\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-oauth-serving-cert\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdw9d\" (UniqueName: \"kubernetes.io/projected/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-kube-api-access-rdw9d\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-config\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-serving-certs-ca-bundle\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.088336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-trusted-ca-bundle\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-oauth-config\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.088911 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.088889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-service-ca\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.089340 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.089320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-oauth-serving-cert\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.089735 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.089716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-config\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.090041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.090020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-trusted-ca-bundle\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.091247 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.091215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-oauth-config\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.091349 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.091314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-serving-cert\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.097647 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.097620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdw9d\" (UniqueName: \"kubernetes.io/projected/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-kube-api-access-rdw9d\") pod \"console-75cc6f7574-9mjmk\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") " pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.188943 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.188893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.188956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-federate-client-tls\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.188987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-secret-telemeter-client\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.189026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.189072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-serving-certs-ca-bundle\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.189238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dtf\" (UniqueName: \"kubernetes.io/projected/c19e5383-0743-4d11-9b1a-fceb9969556a-kube-api-access-k8dtf\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.189285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-metrics-client-ca\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.189411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.189316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-telemeter-client-tls\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.190307 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.190280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-serving-certs-ca-bundle\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.190401 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.190379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.190469 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.190397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19e5383-0743-4d11-9b1a-fceb9969556a-metrics-client-ca\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.192062 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.192028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.192176 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.192109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-telemeter-client-tls\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.192281 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.192256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-federate-client-tls\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.192771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.192703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c19e5383-0743-4d11-9b1a-fceb9969556a-secret-telemeter-client\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.201969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.201906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dtf\" (UniqueName: \"kubernetes.io/projected/c19e5383-0743-4d11-9b1a-fceb9969556a-kube-api-access-k8dtf\") pod \"telemeter-client-764fcd64dc-j4sfl\" (UID: \"c19e5383-0743-4d11-9b1a-fceb9969556a\") " pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:04.217654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.217617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:04.339569 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:04.339529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"
Sep 11 17:40:06.327677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:06.327615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-645b98cdc6-8g2vt"]
Sep 11 17:40:06.332099 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:06.332072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"]
Sep 11 17:40:06.336928 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:06.336895    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1f8b56_129e_4aac_ba8f_6933c3b84139.slice/crio-56d604c785155f008aa062f6b7b11b1a9ff824930ad15eff3c8b522563f1bff9 WatchSource:0}: Error finding container 56d604c785155f008aa062f6b7b11b1a9ff824930ad15eff3c8b522563f1bff9: Status 404 returned error can't find the container with id 56d604c785155f008aa062f6b7b11b1a9ff824930ad15eff3c8b522563f1bff9
Sep 11 17:40:06.339999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:06.339915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-764fcd64dc-j4sfl"]
Sep 11 17:40:06.344438 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:06.344384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19e5383_0743_4d11_9b1a_fceb9969556a.slice/crio-140d84e895fa26ac8f30f37d0335d9c41d9f0438723dd28d168783242a5a04ca WatchSource:0}: Error finding container 140d84e895fa26ac8f30f37d0335d9c41d9f0438723dd28d168783242a5a04ca: Status 404 returned error can't find the container with id 140d84e895fa26ac8f30f37d0335d9c41d9f0438723dd28d168783242a5a04ca
Sep 11 17:40:06.350332 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:06.350303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75cc6f7574-9mjmk"]
Sep 11 17:40:06.353369 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:06.353342    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db935f5_98ca_4928_a0c1_7e1bd59d6cb6.slice/crio-07ba057634ad0f1bbc9181697d5577586c43fde0e452f2c9b4949d6c4158ecd9 WatchSource:0}: Error finding container 07ba057634ad0f1bbc9181697d5577586c43fde0e452f2c9b4949d6c4158ecd9: Status 404 returned error can't find the container with id 07ba057634ad0f1bbc9181697d5577586c43fde0e452f2c9b4949d6c4158ecd9
Sep 11 17:40:07.173247 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.172432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-94tdw" event={"ID":"6e38817a-672b-4e85-ab7c-405f8eb555c5","Type":"ContainerStarted","Data":"1b503c6edd36a3a2d7c06038bd58f73597691c39e893a365601967da092c6bb9"}
Sep 11 17:40:07.175058 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.174985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cc6f7574-9mjmk" event={"ID":"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6","Type":"ContainerStarted","Data":"07ba057634ad0f1bbc9181697d5577586c43fde0e452f2c9b4949d6c4158ecd9"}
Sep 11 17:40:07.177119 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.177052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl" event={"ID":"7c1f8b56-129e-4aac-ba8f-6933c3b84139","Type":"ContainerStarted","Data":"56d604c785155f008aa062f6b7b11b1a9ff824930ad15eff3c8b522563f1bff9"}
Sep 11 17:40:07.180342 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.180265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl" event={"ID":"c19e5383-0743-4d11-9b1a-fceb9969556a","Type":"ContainerStarted","Data":"140d84e895fa26ac8f30f37d0335d9c41d9f0438723dd28d168783242a5a04ca"}
Sep 11 17:40:07.182090 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.182033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt" event={"ID":"d952eb63-5524-4d2b-b9c5-c40ce20966ba","Type":"ContainerStarted","Data":"4b3a2632428c30d261131c2ccd8ba6e7be995dd197989c5644656907a2c25921"}
Sep 11 17:40:07.184048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.183645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9shmh" event={"ID":"88395018-dee0-4289-97a0-d6313aa0ddc5","Type":"ContainerStarted","Data":"5963c38aaddcf1776438eec1559197ea2cd1b08760ed6e7a77d0f7e6386fc650"}
Sep 11 17:40:07.184048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.183907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:40:07.192128 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:07.192070    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-94tdw" podStartSLOduration=67.952490095 podStartE2EDuration="1m14.192051397s" totalImagesPullingTime="6.239561302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:07.191504888 +0000 UTC m=+73.954959608" watchObservedRunningTime="2026-09-11 17:40:07.192051397 +0000 UTC m=+73.955506181"
Sep 11 17:40:11.203989 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.203949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl" event={"ID":"7c1f8b56-129e-4aac-ba8f-6933c3b84139","Type":"ContainerStarted","Data":"7466f10d7dc4280826fed5b7cc52c9a77eb8091c0bf7e619d7ab25920970d601"}
Sep 11 17:40:11.204530 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.204121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:11.207127 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.207097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl" event={"ID":"c19e5383-0743-4d11-9b1a-fceb9969556a","Type":"ContainerStarted","Data":"3188fdcf1082febc6317fd16cb79c64a04fca2cf894facfec54a7eac79cc8158"}
Sep 11 17:40:11.207250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.207153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl" event={"ID":"c19e5383-0743-4d11-9b1a-fceb9969556a","Type":"ContainerStarted","Data":"93a21274e598325ff6cd6b623544bbb09427451629ab3508d43e152a96648304"}
Sep 11 17:40:11.207250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.207181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl" event={"ID":"c19e5383-0743-4d11-9b1a-fceb9969556a","Type":"ContainerStarted","Data":"e1eef50c4f4867da30271bd183eade8216039618c1cecbf55dbea55e058455dc"}
Sep 11 17:40:11.208991 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.208965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt" event={"ID":"d952eb63-5524-4d2b-b9c5-c40ce20966ba","Type":"ContainerStarted","Data":"bcf3a65b59ffde9ffb839388abf0e78134602cade9717fdb3c62f55fb83e2562"}
Sep 11 17:40:11.210382 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.210340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl"
Sep 11 17:40:11.212831 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.212808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"}
Sep 11 17:40:11.212937 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.212861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"}
Sep 11 17:40:11.212937 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.212888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"}
Sep 11 17:40:11.212937 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.212925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"}
Sep 11 17:40:11.213097 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.212960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"}
Sep 11 17:40:11.214524 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.214501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cc6f7574-9mjmk" event={"ID":"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6","Type":"ContainerStarted","Data":"7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a"}
Sep 11 17:40:11.234476 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.234424    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q7pnl" podStartSLOduration=4.415018603 podStartE2EDuration="8.23440786s" totalImagesPullingTime="3.819389257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:11.233366926 +0000 UTC m=+77.996821656" watchObservedRunningTime="2026-09-11 17:40:11.23440786 +0000 UTC m=+77.997862587"
Sep 11 17:40:11.234658 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.234547    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9shmh" podStartSLOduration=72.582376368 podStartE2EDuration="1m18.234539201s" totalImagesPullingTime="5.652162833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:07.210235236 +0000 UTC m=+73.973689963" watchObservedRunningTime="2026-09-11 17:40:11.234539201 +0000 UTC m=+77.997993931"
Sep 11 17:40:11.355259 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.355208    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-764fcd64dc-j4sfl" podStartSLOduration=3.536009451 podStartE2EDuration="7.35518938s" totalImagesPullingTime="3.819179929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:11.302357234 +0000 UTC m=+78.065811960" watchObservedRunningTime="2026-09-11 17:40:11.35518938 +0000 UTC m=+78.118644107"
Sep 11 17:40:11.381325 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.381275    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt" podStartSLOduration=4.555566789 podStartE2EDuration="8.381256064s" totalImagesPullingTime="3.825689275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:11.354029339 +0000 UTC m=+78.117484066" watchObservedRunningTime="2026-09-11 17:40:11.381256064 +0000 UTC m=+78.144710791"
Sep 11 17:40:11.381463 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:11.381362    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75cc6f7574-9mjmk" podStartSLOduration=4.568077981 podStartE2EDuration="8.381358415s" totalImagesPullingTime="3.813280434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:11.380132861 +0000 UTC m=+78.143587612" watchObservedRunningTime="2026-09-11 17:40:11.381358415 +0000 UTC m=+78.144813142"
Sep 11 17:40:12.222848 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:12.222809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerStarted","Data":"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"}
Sep 11 17:40:12.260583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:12.260530    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.1598326979999998 podStartE2EDuration="13.260513336s" totalImagesPullingTime="10.078565771s" totalInitContainerRuntime="22.114867ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:12.257889002 +0000 UTC m=+79.021343778" watchObservedRunningTime="2026-09-11 17:40:12.260513336 +0000 UTC m=+79.023968062"
Sep 11 17:40:12.930233 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:12.930196    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75cc6f7574-9mjmk"]
Sep 11 17:40:12.963105 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:12.963071    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8698bd8cff-dbt4q"]
Sep 11 17:40:12.966142 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:12.966123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:12.987851 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:12.987827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8698bd8cff-dbt4q"]
Sep 11 17:40:13.073334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj25b\" (UniqueName: \"kubernetes.io/projected/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-kube-api-access-vj25b\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.073542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-oauth-serving-cert\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.073542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-config\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.073542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-oauth-config\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.073542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-trusted-ca-bundle\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.073542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-serving-cert\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.073760 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.073571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-service-ca\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.174998 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.174964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj25b\" (UniqueName: \"kubernetes.io/projected/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-kube-api-access-vj25b\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.174998 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-oauth-serving-cert\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-config\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-oauth-config\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-trusted-ca-bundle\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-serving-cert\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-service-ca\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175879 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-oauth-serving-cert\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.175879 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.175848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-config\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.176057 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.176030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-trusted-ca-bundle\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.176138 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.176044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-service-ca\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.177842 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.177822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-serving-cert\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.177940 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.177845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-oauth-config\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.185056 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.184999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj25b\" (UniqueName: \"kubernetes.io/projected/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-kube-api-access-vj25b\") pod \"console-8698bd8cff-dbt4q\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") " pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.275132 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.275097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:13.550515 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:13.550470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8698bd8cff-dbt4q"]
Sep 11 17:40:13.553035 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:40:13.553008    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c28cd4_f47b_487e_a209_b8c4dc6307ba.slice/crio-abaf5281c1a07ce67754f3d7021f78911c97bbb750ef11609a772020e7b07132 WatchSource:0}: Error finding container abaf5281c1a07ce67754f3d7021f78911c97bbb750ef11609a772020e7b07132: Status 404 returned error can't find the container with id abaf5281c1a07ce67754f3d7021f78911c97bbb750ef11609a772020e7b07132
Sep 11 17:40:14.218821 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:14.218791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:14.232772 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:14.232732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698bd8cff-dbt4q" event={"ID":"f4c28cd4-f47b-487e-a209-b8c4dc6307ba","Type":"ContainerStarted","Data":"8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd"}
Sep 11 17:40:14.232933 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:14.232780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698bd8cff-dbt4q" event={"ID":"f4c28cd4-f47b-487e-a209-b8c4dc6307ba","Type":"ContainerStarted","Data":"abaf5281c1a07ce67754f3d7021f78911c97bbb750ef11609a772020e7b07132"}
Sep 11 17:40:14.254290 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:14.254242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8698bd8cff-dbt4q" podStartSLOduration=2.25422608 podStartE2EDuration="2.25422608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:14.253048115 +0000 UTC m=+81.016502842" watchObservedRunningTime="2026-09-11 17:40:14.25422608 +0000 UTC m=+81.017680807"
Sep 11 17:40:23.275503 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:23.275463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:23.275935 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:23.275560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:23.280525 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:23.280480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:23.750405 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:23.750373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:23.750602 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:23.750429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:24.275635 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:24.275607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:40:38.192957 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.192929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9shmh"
Sep 11 17:40:38.246752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.246689    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75cc6f7574-9mjmk" podUID="8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" containerName="console" containerID="cri-o://7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a" gracePeriod=15
Sep 11 17:40:38.525940 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.525905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:38.691868 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.691831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-service-ca\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692032 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.691893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdw9d\" (UniqueName: \"kubernetes.io/projected/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-kube-api-access-rdw9d\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692032 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.691931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-oauth-config\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692032 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.691963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-serving-cert\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692032 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.691982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-config\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692252 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.692034    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-oauth-serving-cert\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692252 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.692068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-trusted-ca-bundle\") pod \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\" (UID: \"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6\") "
Sep 11 17:40:38.692388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.692351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-service-ca" (OuterVolumeSpecName: "service-ca") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:38.692561 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.692464    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-config" (OuterVolumeSpecName: "console-config") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:38.692561 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.692482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:38.692680 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.692599    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:38.694592 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.694558    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:40:38.694592 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.694583    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-kube-api-access-rdw9d" (OuterVolumeSpecName: "kube-api-access-rdw9d") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "kube-api-access-rdw9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:40:38.694744 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.694621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" (UID: "8db935f5-98ca-4928-a0c1-7e1bd59d6cb6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793264    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-oauth-serving-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793297    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-trusted-ca-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793307    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793316    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdw9d\" (UniqueName: \"kubernetes.io/projected/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-kube-api-access-rdw9d\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793325    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-oauth-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793334    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-serving-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:38.793351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:38.793341    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6-console-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:40:39.328884 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.328848    2563 generic.go:356] "Generic (PLEG): container finished" podID="8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" containerID="7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a" exitCode=0
Sep 11 17:40:39.329264 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.328895    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75cc6f7574-9mjmk"
Sep 11 17:40:39.329264 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.328933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cc6f7574-9mjmk" event={"ID":"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6","Type":"ContainerDied","Data":"7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a"}
Sep 11 17:40:39.329264 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.328985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75cc6f7574-9mjmk" event={"ID":"8db935f5-98ca-4928-a0c1-7e1bd59d6cb6","Type":"ContainerDied","Data":"07ba057634ad0f1bbc9181697d5577586c43fde0e452f2c9b4949d6c4158ecd9"}
Sep 11 17:40:39.329264 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.329005    2563 scope.go:117] "RemoveContainer" containerID="7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a"
Sep 11 17:40:39.337880 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.337864    2563 scope.go:117] "RemoveContainer" containerID="7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a"
Sep 11 17:40:39.338130 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:40:39.338113    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a\": container with ID starting with 7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a not found: ID does not exist" containerID="7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a"
Sep 11 17:40:39.338174 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.338138    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a"} err="failed to get container status \"7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a\": rpc error: code = NotFound desc = could not find container \"7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a\": container with ID starting with 7755c9f5db715b058e597b004f53e2acc10ca58cf032c06088bd2b0129eda31a not found: ID does not exist"
Sep 11 17:40:39.355151 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.355124    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75cc6f7574-9mjmk"]
Sep 11 17:40:39.356157 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.356140    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75cc6f7574-9mjmk"]
Sep 11 17:40:39.779844 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:39.779770    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" path="/var/lib/kubelet/pods/8db935f5-98ca-4928-a0c1-7e1bd59d6cb6/volumes"
Sep 11 17:40:43.755356 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:43.755328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:40:43.759177 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:40:43.759161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-645b98cdc6-8g2vt"
Sep 11 17:41:19.212575 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.212540    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:41:19.213095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.212818    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="alertmanager" containerID="cri-o://fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28" gracePeriod=120
Sep 11 17:41:19.213095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.212900    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-web" containerID="cri-o://0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64" gracePeriod=120
Sep 11 17:41:19.213095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.212893    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-metric" containerID="cri-o://3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320" gracePeriod=120
Sep 11 17:41:19.213095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.212980    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy" containerID="cri-o://d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185" gracePeriod=120
Sep 11 17:41:19.213095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.212955    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="config-reloader" containerID="cri-o://86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1" gracePeriod=120
Sep 11 17:41:19.213095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.213081    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="prom-label-proxy" containerID="cri-o://d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9" gracePeriod=120
Sep 11 17:41:19.478161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478067    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9" exitCode=0
Sep 11 17:41:19.478161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478101    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185" exitCode=0
Sep 11 17:41:19.478161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478108    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1" exitCode=0
Sep 11 17:41:19.478161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478115    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28" exitCode=0
Sep 11 17:41:19.478161 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"}
Sep 11 17:41:19.478425 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"}
Sep 11 17:41:19.478425 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"}
Sep 11 17:41:19.478425 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:19.478260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"}
Sep 11 17:41:20.453334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.453311    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.485760 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485731    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320" exitCode=0
Sep 11 17:41:20.485760 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485755    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerID="0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64" exitCode=0
Sep 11 17:41:20.485958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.485958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"}
Sep 11 17:41:20.485958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"}
Sep 11 17:41:20.485958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00","Type":"ContainerDied","Data":"487ac353bc3b51680b2aa70fa884a9d8b56d4456004162a0b9337b5b654c266e"}
Sep 11 17:41:20.486083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.485965    2563 scope.go:117] "RemoveContainer" containerID="d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"
Sep 11 17:41:20.495110 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.495092    2563 scope.go:117] "RemoveContainer" containerID="3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"
Sep 11 17:41:20.502987 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.502968    2563 scope.go:117] "RemoveContainer" containerID="d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"
Sep 11 17:41:20.515601 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.515578    2563 scope.go:117] "RemoveContainer" containerID="0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"
Sep 11 17:41:20.524145 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524112    2563 scope.go:117] "RemoveContainer" containerID="86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"
Sep 11 17:41:20.524277 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-main-tls\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524341 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-tls-assets\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524341 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-web-config\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-cluster-tls-config\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-out\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb4g7\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-kube-api-access-jb4g7\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-metrics-client-ca\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-main-db\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-volume\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-web\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.524806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.524623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-trusted-ca-bundle\") pod \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\" (UID: \"7e0e684e-f6b7-403b-8ca8-f7e67afd1c00\") "
Sep 11 17:41:20.525549 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.525104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:20.525549 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.525389    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:41:20.526423 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.526291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:41:20.528571 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.528532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.528774 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.528722    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.528944 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.528906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.528944 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.528933    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:41:20.529416 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.529359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-out" (OuterVolumeSpecName: "config-out") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:20.530658 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.530616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-volume" (OuterVolumeSpecName: "config-volume") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.530766 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.530714    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.530766 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.530720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-kube-api-access-jb4g7" (OuterVolumeSpecName: "kube-api-access-jb4g7") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "kube-api-access-jb4g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:41:20.533900 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.533799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.541334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.541309    2563 scope.go:117] "RemoveContainer" containerID="fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"
Sep 11 17:41:20.542012 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.541987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-web-config" (OuterVolumeSpecName: "web-config") pod "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" (UID: "7e0e684e-f6b7-403b-8ca8-f7e67afd1c00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:41:20.549091 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.549071    2563 scope.go:117] "RemoveContainer" containerID="3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"
Sep 11 17:41:20.556808 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.556778    2563 scope.go:117] "RemoveContainer" containerID="d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"
Sep 11 17:41:20.557126 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.557097    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9\": container with ID starting with d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9 not found: ID does not exist" containerID="d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"
Sep 11 17:41:20.557226 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.557135    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"} err="failed to get container status \"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9\": rpc error: code = NotFound desc = could not find container \"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9\": container with ID starting with d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9 not found: ID does not exist"
Sep 11 17:41:20.557226 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.557155    2563 scope.go:117] "RemoveContainer" containerID="3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"
Sep 11 17:41:20.557464 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.557450    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320\": container with ID starting with 3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320 not found: ID does not exist" containerID="3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"
Sep 11 17:41:20.557545 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.557472    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"} err="failed to get container status \"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320\": rpc error: code = NotFound desc = could not find container \"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320\": container with ID starting with 3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320 not found: ID does not exist"
Sep 11 17:41:20.557545 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.557502    2563 scope.go:117] "RemoveContainer" containerID="d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"
Sep 11 17:41:20.557762 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.557733    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185\": container with ID starting with d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185 not found: ID does not exist" containerID="d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"
Sep 11 17:41:20.557850 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.557764    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"} err="failed to get container status \"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185\": rpc error: code = NotFound desc = could not find container \"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185\": container with ID starting with d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185 not found: ID does not exist"
Sep 11 17:41:20.557850 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.557777    2563 scope.go:117] "RemoveContainer" containerID="0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"
Sep 11 17:41:20.558052 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.558028    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64\": container with ID starting with 0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64 not found: ID does not exist" containerID="0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"
Sep 11 17:41:20.558095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558062    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"} err="failed to get container status \"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64\": rpc error: code = NotFound desc = could not find container \"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64\": container with ID starting with 0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64 not found: ID does not exist"
Sep 11 17:41:20.558095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558085    2563 scope.go:117] "RemoveContainer" containerID="86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"
Sep 11 17:41:20.558380 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.558363    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1\": container with ID starting with 86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1 not found: ID does not exist" containerID="86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"
Sep 11 17:41:20.558437 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558386    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"} err="failed to get container status \"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1\": rpc error: code = NotFound desc = could not find container \"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1\": container with ID starting with 86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1 not found: ID does not exist"
Sep 11 17:41:20.558437 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558403    2563 scope.go:117] "RemoveContainer" containerID="fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"
Sep 11 17:41:20.558644 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.558628    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28\": container with ID starting with fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28 not found: ID does not exist" containerID="fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"
Sep 11 17:41:20.558692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558650    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"} err="failed to get container status \"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28\": rpc error: code = NotFound desc = could not find container \"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28\": container with ID starting with fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28 not found: ID does not exist"
Sep 11 17:41:20.558692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558667    2563 scope.go:117] "RemoveContainer" containerID="3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"
Sep 11 17:41:20.558893 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:41:20.558877    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5\": container with ID starting with 3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5 not found: ID does not exist" containerID="3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"
Sep 11 17:41:20.558947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558899    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"} err="failed to get container status \"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5\": rpc error: code = NotFound desc = could not find container \"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5\": container with ID starting with 3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5 not found: ID does not exist"
Sep 11 17:41:20.558947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.558916    2563 scope.go:117] "RemoveContainer" containerID="d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"
Sep 11 17:41:20.559143 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559125    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9"} err="failed to get container status \"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9\": rpc error: code = NotFound desc = could not find container \"d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9\": container with ID starting with d3887ed36d7d67311fadc938eebbf8c20ae4fd1ca50cf741a77decb8da6a40a9 not found: ID does not exist"
Sep 11 17:41:20.559189 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559144    2563 scope.go:117] "RemoveContainer" containerID="3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"
Sep 11 17:41:20.559398 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559374    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320"} err="failed to get container status \"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320\": rpc error: code = NotFound desc = could not find container \"3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320\": container with ID starting with 3809501b9a0223ed817f6b3324a88d5a1fab9bf33507989e9232ac872c454320 not found: ID does not exist"
Sep 11 17:41:20.559453 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559401    2563 scope.go:117] "RemoveContainer" containerID="d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"
Sep 11 17:41:20.559697 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559676    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185"} err="failed to get container status \"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185\": rpc error: code = NotFound desc = could not find container \"d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185\": container with ID starting with d28ab38fa170912b8ffc32591e3cfa9de4ec7b44ca8281800d72cc43b9193185 not found: ID does not exist"
Sep 11 17:41:20.559771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559698    2563 scope.go:117] "RemoveContainer" containerID="0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"
Sep 11 17:41:20.559979 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559960    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64"} err="failed to get container status \"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64\": rpc error: code = NotFound desc = could not find container \"0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64\": container with ID starting with 0e337590c101c7ebe7cd2aca5dd1426bfc3ee1beb104e6f73e6731fce7f26a64 not found: ID does not exist"
Sep 11 17:41:20.560059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.559980    2563 scope.go:117] "RemoveContainer" containerID="86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"
Sep 11 17:41:20.560230 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.560214    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1"} err="failed to get container status \"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1\": rpc error: code = NotFound desc = could not find container \"86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1\": container with ID starting with 86b3b2f2e5d43e59720fe14e2b3c2d182bd683ca57eed39543631ccd4d79b1e1 not found: ID does not exist"
Sep 11 17:41:20.560230 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.560230    2563 scope.go:117] "RemoveContainer" containerID="fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"
Sep 11 17:41:20.560518 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.560503    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28"} err="failed to get container status \"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28\": rpc error: code = NotFound desc = could not find container \"fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28\": container with ID starting with fbd387dcccdc798dce6109098691f366fc06fc91d867ee955271335a044bca28 not found: ID does not exist"
Sep 11 17:41:20.560579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.560518    2563 scope.go:117] "RemoveContainer" containerID="3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"
Sep 11 17:41:20.560760 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.560742    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5"} err="failed to get container status \"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5\": rpc error: code = NotFound desc = could not find container \"3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5\": container with ID starting with 3f0ea5b794c6de07e9e7021f84ea32cc3ebb9c8009c510b247fa6e5d6e0326f5 not found: ID does not exist"
Sep 11 17:41:20.625681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625642    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4g7\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-kube-api-access-jb4g7\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625676    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-metrics-client-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625687    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-main-db\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625696    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625706    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625715    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625724    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625733    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625743    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-secret-alertmanager-main-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625751    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-tls-assets\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625760    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-web-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625767    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-cluster-tls-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.625898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.625774    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00-config-out\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:41:20.810421 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.810386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:41:20.816275 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.816249    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:41:20.849001 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.848971    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:41:20.849292 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849277    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-metric"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849297    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-metric"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849309    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="prom-label-proxy"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849343    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="prom-label-proxy"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849355    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="alertmanager"
Sep 11 17:41:20.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849364    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="alertmanager"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849378    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="config-reloader"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849387    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="config-reloader"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849399    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="init-config-reloader"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849407    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="init-config-reloader"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849420    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-web"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849429    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-web"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" containerName="console"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849448    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" containerName="console"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849533    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db935f5-98ca-4928-a0c1-7e1bd59d6cb6" containerName="console"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849545    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-web"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849555    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849563    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="prom-label-proxy"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849573    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="config-reloader"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849581    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="kube-rbac-proxy-metric"
Sep 11 17:41:20.849745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.849589    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" containerName="alertmanager"
Sep 11 17:41:20.853263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.853243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.856351 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856330    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:41:20.856532 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:41:20.856618 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856595    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:41:20.856679 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:41:20.856679 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:41:20.856892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:41:20.856963 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.856939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jg7mx\""
Sep 11 17:41:20.857038 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.857024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:41:20.857115 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.857098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:41:20.866762 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.866732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:41:20.873073 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.873049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:41:20.927776 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.927736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.927958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.927792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.927958 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.927822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.927957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928166 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928211 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928297 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-web-config\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928429 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-config-out\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:20.928429 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:20.928394    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzf6x\" (UniqueName: \"kubernetes.io/projected/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-kube-api-access-qzf6x\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-web-config\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.029727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-config-out\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.030052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzf6x\" (UniqueName: \"kubernetes.io/projected/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-kube-api-access-qzf6x\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.030052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.030052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.030052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.030052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.029863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.031637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.031318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.031637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.031346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.031637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.031587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.032953 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.032925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.033080 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.032962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.033143 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.033098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-web-config\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.033306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.033283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.033480 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.033462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-config-out\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.033592 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.033548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.033592 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.033555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-config-volume\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.034593 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.034577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.034962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.034946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.038409 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.038393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzf6x\" (UniqueName: \"kubernetes.io/projected/c9961fae-dd3b-4e48-a2e1-ea00a01c33db-kube-api-access-qzf6x\") pod \"alertmanager-main-0\" (UID: \"c9961fae-dd3b-4e48-a2e1-ea00a01c33db\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.163020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.162978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:41:21.446393 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.446368    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:41:21.448735 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:41:21.448695    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9961fae_dd3b_4e48_a2e1_ea00a01c33db.slice/crio-bf18ca5be001f4c3ba5afd2c4d9de59df5946551dcd2fac32d488adc17bd2910 WatchSource:0}: Error finding container bf18ca5be001f4c3ba5afd2c4d9de59df5946551dcd2fac32d488adc17bd2910: Status 404 returned error can't find the container with id bf18ca5be001f4c3ba5afd2c4d9de59df5946551dcd2fac32d488adc17bd2910
Sep 11 17:41:21.490973 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.490935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"bf18ca5be001f4c3ba5afd2c4d9de59df5946551dcd2fac32d488adc17bd2910"}
Sep 11 17:41:21.780085 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:21.779981    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0e684e-f6b7-403b-8ca8-f7e67afd1c00" path="/var/lib/kubelet/pods/7e0e684e-f6b7-403b-8ca8-f7e67afd1c00/volumes"
Sep 11 17:41:22.496829 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:22.496797    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9961fae-dd3b-4e48-a2e1-ea00a01c33db" containerID="5d2c41bbcab168cacc312c4e7892da3b1a20b6a040a06414c06cb5d842825b9a" exitCode=0
Sep 11 17:41:22.497310 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:22.496842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerDied","Data":"5d2c41bbcab168cacc312c4e7892da3b1a20b6a040a06414c06cb5d842825b9a"}
Sep 11 17:41:23.506267 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.506235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"cd907e438c895f237a7c7541eeb60b27bc8b040e031ab14b7fcc1fef11e95e92"}
Sep 11 17:41:23.506796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.506315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"b08e9bd3bce3561d0e770abb938fcc0e34c83fe2165c60dd9769addc3cb3874b"}
Sep 11 17:41:23.506796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.506354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"cbd877f120b3d09bde5ca3b4e1e700e3b9904477be26323bbc17dbabd4620775"}
Sep 11 17:41:23.506796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.506396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"964a8619c39a6c841273a1cc8c44eee3a995b850395d72c26ac51cc18aff38fe"}
Sep 11 17:41:23.506796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.506438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"584bdfa0bb675f98bb7660263ff3c633bde304b13f859a7432257fae9469b60e"}
Sep 11 17:41:23.506796 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.506475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c9961fae-dd3b-4e48-a2e1-ea00a01c33db","Type":"ContainerStarted","Data":"112da525f1e6cf136b4dd016a428fa7b12121885accea1069ca616528da0b76a"}
Sep 11 17:41:23.555021 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:23.554963    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.539289992 podStartE2EDuration="3.55494538s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.655388ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:41:23.553853311 +0000 UTC m=+150.317308037" watchObservedRunningTime="2026-09-11 17:41:23.55494538 +0000 UTC m=+150.318400121"
Sep 11 17:41:43.653198 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.653165    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679c4568b5-jjdv7"]
Sep 11 17:41:43.656212 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.656188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.687042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.687010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679c4568b5-jjdv7"]
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-service-ca\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-oauth-config\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-oauth-serving-cert\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp9d\" (UniqueName: \"kubernetes.io/projected/ed7597b1-62b0-4131-bee6-d9564147bc9f-kube-api-access-lhp9d\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-config\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-serving-cert\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.722583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.719655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-trusted-ca-bundle\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820418 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp9d\" (UniqueName: \"kubernetes.io/projected/ed7597b1-62b0-4131-bee6-d9564147bc9f-kube-api-access-lhp9d\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820418 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-config\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-serving-cert\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-trusted-ca-bundle\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-service-ca\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-oauth-config\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.820700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.820563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-oauth-serving-cert\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.821417 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.821395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-oauth-serving-cert\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.821553 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.821477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-trusted-ca-bundle\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.821553 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.821529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-config\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.821632 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.821594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-service-ca\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.823091 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.823071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-oauth-config\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.823215 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.823198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-serving-cert\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.829153 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.829130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp9d\" (UniqueName: \"kubernetes.io/projected/ed7597b1-62b0-4131-bee6-d9564147bc9f-kube-api-access-lhp9d\") pod \"console-679c4568b5-jjdv7\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") " pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:43.965436 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:43.965338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:44.240549 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:44.240441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679c4568b5-jjdv7"]
Sep 11 17:41:44.243390 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:41:44.243358    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7597b1_62b0_4131_bee6_d9564147bc9f.slice/crio-3571c2daa3304b80025181682e9b1710a9b80ef8aed6be5426c956e3a5b7d1de WatchSource:0}: Error finding container 3571c2daa3304b80025181682e9b1710a9b80ef8aed6be5426c956e3a5b7d1de: Status 404 returned error can't find the container with id 3571c2daa3304b80025181682e9b1710a9b80ef8aed6be5426c956e3a5b7d1de
Sep 11 17:41:44.585771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:44.585739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679c4568b5-jjdv7" event={"ID":"ed7597b1-62b0-4131-bee6-d9564147bc9f","Type":"ContainerStarted","Data":"ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7"}
Sep 11 17:41:44.585944 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:44.585787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679c4568b5-jjdv7" event={"ID":"ed7597b1-62b0-4131-bee6-d9564147bc9f","Type":"ContainerStarted","Data":"3571c2daa3304b80025181682e9b1710a9b80ef8aed6be5426c956e3a5b7d1de"}
Sep 11 17:41:44.606296 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:44.606244    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679c4568b5-jjdv7" podStartSLOduration=1.606226039 podStartE2EDuration="1.606226039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:41:44.605827886 +0000 UTC m=+171.369282614" watchObservedRunningTime="2026-09-11 17:41:44.606226039 +0000 UTC m=+171.369680766"
Sep 11 17:41:53.966020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:53.965923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:53.966020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:53.965990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:53.970891 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:53.970871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:54.627002 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:54.626973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:41:54.677549 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:41:54.677514    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8698bd8cff-dbt4q"]
Sep 11 17:42:19.696674 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:19.696625    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8698bd8cff-dbt4q" podUID="f4c28cd4-f47b-487e-a209-b8c4dc6307ba" containerName="console" containerID="cri-o://8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd" gracePeriod=15
Sep 11 17:42:19.937735 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:19.937712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:42:20.012075 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.011972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-oauth-config\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012075 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj25b\" (UniqueName: \"kubernetes.io/projected/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-kube-api-access-vj25b\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012086    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-service-ca\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-oauth-serving-cert\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-trusted-ca-bundle\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-config\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-serving-cert\") pod \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\" (UID: \"f4c28cd4-f47b-487e-a209-b8c4dc6307ba\") "
Sep 11 17:42:20.012574 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:42:20.012642 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:42:20.012642 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-config" (OuterVolumeSpecName: "console-config") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:42:20.012749 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.012635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:42:20.014391 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.014365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:42:20.014522 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.014416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-kube-api-access-vj25b" (OuterVolumeSpecName: "kube-api-access-vj25b") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "kube-api-access-vj25b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:42:20.014522 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.014416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4c28cd4-f47b-487e-a209-b8c4dc6307ba" (UID: "f4c28cd4-f47b-487e-a209-b8c4dc6307ba"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:42:20.113309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113258    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj25b\" (UniqueName: \"kubernetes.io/projected/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-kube-api-access-vj25b\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.113309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113302    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.113309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113311    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-oauth-serving-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.113309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113320    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-trusted-ca-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.113309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113328    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.113614 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113336    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-serving-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.113614 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.113344    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4c28cd4-f47b-487e-a209-b8c4dc6307ba-console-oauth-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.723606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.723570    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4c28cd4-f47b-487e-a209-b8c4dc6307ba" containerID="8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd" exitCode=0
Sep 11 17:42:20.724042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.723619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698bd8cff-dbt4q"
Sep 11 17:42:20.724042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.723650    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698bd8cff-dbt4q" event={"ID":"f4c28cd4-f47b-487e-a209-b8c4dc6307ba","Type":"ContainerDied","Data":"8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd"}
Sep 11 17:42:20.724042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.723706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698bd8cff-dbt4q" event={"ID":"f4c28cd4-f47b-487e-a209-b8c4dc6307ba","Type":"ContainerDied","Data":"abaf5281c1a07ce67754f3d7021f78911c97bbb750ef11609a772020e7b07132"}
Sep 11 17:42:20.724042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.723728    2563 scope.go:117] "RemoveContainer" containerID="8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd"
Sep 11 17:42:20.732307 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.732290    2563 scope.go:117] "RemoveContainer" containerID="8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd"
Sep 11 17:42:20.732635 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:42:20.732617    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd\": container with ID starting with 8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd not found: ID does not exist" containerID="8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd"
Sep 11 17:42:20.732678 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.732645    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd"} err="failed to get container status \"8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd\": rpc error: code = NotFound desc = could not find container \"8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd\": container with ID starting with 8033d675683e8fda50c9c464f728c140a41dc41aed9c3b0683f79afe21d3e6bd not found: ID does not exist"
Sep 11 17:42:20.745008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.744981    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8698bd8cff-dbt4q"]
Sep 11 17:42:20.749838 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:20.749814    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8698bd8cff-dbt4q"]
Sep 11 17:42:21.780556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:21.780480    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c28cd4-f47b-487e-a209-b8c4dc6307ba" path="/var/lib/kubelet/pods/f4c28cd4-f47b-487e-a209-b8c4dc6307ba/volumes"
Sep 11 17:42:29.411646 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.411612    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"]
Sep 11 17:42:29.412094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.411897    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c28cd4-f47b-487e-a209-b8c4dc6307ba" containerName="console"
Sep 11 17:42:29.412094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.411908    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c28cd4-f47b-487e-a209-b8c4dc6307ba" containerName="console"
Sep 11 17:42:29.412094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.411963    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c28cd4-f47b-487e-a209-b8c4dc6307ba" containerName="console"
Sep 11 17:42:29.416300 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.416284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.419229 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.419208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:42:29.419353 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.419312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z7qc9\""
Sep 11 17:42:29.420702 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.420687    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:42:29.424133 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.424100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"]
Sep 11 17:42:29.489932 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.489897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.489932 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.489936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.490164 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.489993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9w8\" (UniqueName: \"kubernetes.io/projected/eb7f70a3-985c-497e-9314-b02359eb64e0-kube-api-access-rs9w8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.590653 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.590617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9w8\" (UniqueName: \"kubernetes.io/projected/eb7f70a3-985c-497e-9314-b02359eb64e0-kube-api-access-rs9w8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.590843 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.590674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.590843 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.590701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.591049 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.591032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.591127 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.591099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.599712 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.599676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9w8\" (UniqueName: \"kubernetes.io/projected/eb7f70a3-985c-497e-9314-b02359eb64e0-kube-api-access-rs9w8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:29.726811 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:29.726708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:30.015506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:30.015416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"]
Sep 11 17:42:30.018603 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:42:30.018571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7f70a3_985c_497e_9314_b02359eb64e0.slice/crio-087e70ecaf02804609a6855bfc4b0cf7e863049dd03760433da82af07f36467b WatchSource:0}: Error finding container 087e70ecaf02804609a6855bfc4b0cf7e863049dd03760433da82af07f36467b: Status 404 returned error can't find the container with id 087e70ecaf02804609a6855bfc4b0cf7e863049dd03760433da82af07f36467b
Sep 11 17:42:30.766595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:30.766552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv" event={"ID":"eb7f70a3-985c-497e-9314-b02359eb64e0","Type":"ContainerStarted","Data":"087e70ecaf02804609a6855bfc4b0cf7e863049dd03760433da82af07f36467b"}
Sep 11 17:42:35.789212 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:35.789129    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerID="e8e89ba005825bcfd8f44f39defcf5de4a944792b1bb57e9a58a04a8f763d5a1" exitCode=0
Sep 11 17:42:35.789212 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:35.789167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv" event={"ID":"eb7f70a3-985c-497e-9314-b02359eb64e0","Type":"ContainerDied","Data":"e8e89ba005825bcfd8f44f39defcf5de4a944792b1bb57e9a58a04a8f763d5a1"}
Sep 11 17:42:40.809774 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:40.809737    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerID="ba2218665512cf21c82ec2e3aaf71b3d172b85179ebaeed97e95027befe8bb7b" exitCode=0
Sep 11 17:42:40.810158 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:40.809828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv" event={"ID":"eb7f70a3-985c-497e-9314-b02359eb64e0","Type":"ContainerDied","Data":"ba2218665512cf21c82ec2e3aaf71b3d172b85179ebaeed97e95027befe8bb7b"}
Sep 11 17:42:48.845805 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:48.845768    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerID="b86571185a4b8007379707f7085760456da60004828a661bc246c3eb73f490d2" exitCode=0
Sep 11 17:42:48.846199 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:48.845853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv" event={"ID":"eb7f70a3-985c-497e-9314-b02359eb64e0","Type":"ContainerDied","Data":"b86571185a4b8007379707f7085760456da60004828a661bc246c3eb73f490d2"}
Sep 11 17:42:49.991031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:49.991003    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:50.069736 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.069704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-bundle\") pod \"eb7f70a3-985c-497e-9314-b02359eb64e0\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") "
Sep 11 17:42:50.069895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.069747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs9w8\" (UniqueName: \"kubernetes.io/projected/eb7f70a3-985c-497e-9314-b02359eb64e0-kube-api-access-rs9w8\") pod \"eb7f70a3-985c-497e-9314-b02359eb64e0\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") "
Sep 11 17:42:50.069895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.069790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-util\") pod \"eb7f70a3-985c-497e-9314-b02359eb64e0\" (UID: \"eb7f70a3-985c-497e-9314-b02359eb64e0\") "
Sep 11 17:42:50.070369 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.070338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-bundle" (OuterVolumeSpecName: "bundle") pod "eb7f70a3-985c-497e-9314-b02359eb64e0" (UID: "eb7f70a3-985c-497e-9314-b02359eb64e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:50.072085 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.072053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7f70a3-985c-497e-9314-b02359eb64e0-kube-api-access-rs9w8" (OuterVolumeSpecName: "kube-api-access-rs9w8") pod "eb7f70a3-985c-497e-9314-b02359eb64e0" (UID: "eb7f70a3-985c-497e-9314-b02359eb64e0"). InnerVolumeSpecName "kube-api-access-rs9w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:42:50.074107 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.074088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-util" (OuterVolumeSpecName: "util") pod "eb7f70a3-985c-497e-9314-b02359eb64e0" (UID: "eb7f70a3-985c-497e-9314-b02359eb64e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:50.170578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.170474    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:50.170578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.170525    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb7f70a3-985c-497e-9314-b02359eb64e0-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:50.170578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.170535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9w8\" (UniqueName: \"kubernetes.io/projected/eb7f70a3-985c-497e-9314-b02359eb64e0-kube-api-access-rs9w8\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:42:50.855598 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.855562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv" event={"ID":"eb7f70a3-985c-497e-9314-b02359eb64e0","Type":"ContainerDied","Data":"087e70ecaf02804609a6855bfc4b0cf7e863049dd03760433da82af07f36467b"}
Sep 11 17:42:50.855826 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.855621    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087e70ecaf02804609a6855bfc4b0cf7e863049dd03760433da82af07f36467b"
Sep 11 17:42:50.855826 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:50.855574    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sccxv"
Sep 11 17:42:57.438365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438333    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"]
Sep 11 17:42:57.438741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="util"
Sep 11 17:42:57.438741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438652    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="util"
Sep 11 17:42:57.438741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438662    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="pull"
Sep 11 17:42:57.438741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438667    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="pull"
Sep 11 17:42:57.438741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438701    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="extract"
Sep 11 17:42:57.438741 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438708    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="extract"
Sep 11 17:42:57.438924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.438748    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7f70a3-985c-497e-9314-b02359eb64e0" containerName="extract"
Sep 11 17:42:57.441265 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.441249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.444104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.444086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:42:57.444363 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.444345    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vdqdj\""
Sep 11 17:42:57.444442 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.444349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 17:42:57.452524 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.452480    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"]
Sep 11 17:42:57.527706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.527668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfhq\" (UniqueName: \"kubernetes.io/projected/3b7d9194-df82-4f00-9eb0-2a1df12fa34a-kube-api-access-gcfhq\") pod \"cert-manager-operator-controller-manager-55c444bf86-45kts\" (UID: \"3b7d9194-df82-4f00-9eb0-2a1df12fa34a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.527706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.527706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b7d9194-df82-4f00-9eb0-2a1df12fa34a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-45kts\" (UID: \"3b7d9194-df82-4f00-9eb0-2a1df12fa34a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.628275 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.628236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfhq\" (UniqueName: \"kubernetes.io/projected/3b7d9194-df82-4f00-9eb0-2a1df12fa34a-kube-api-access-gcfhq\") pod \"cert-manager-operator-controller-manager-55c444bf86-45kts\" (UID: \"3b7d9194-df82-4f00-9eb0-2a1df12fa34a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.628275 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.628276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b7d9194-df82-4f00-9eb0-2a1df12fa34a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-45kts\" (UID: \"3b7d9194-df82-4f00-9eb0-2a1df12fa34a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.628732 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.628715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b7d9194-df82-4f00-9eb0-2a1df12fa34a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-45kts\" (UID: \"3b7d9194-df82-4f00-9eb0-2a1df12fa34a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.636804 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.636781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfhq\" (UniqueName: \"kubernetes.io/projected/3b7d9194-df82-4f00-9eb0-2a1df12fa34a-kube-api-access-gcfhq\") pod \"cert-manager-operator-controller-manager-55c444bf86-45kts\" (UID: \"3b7d9194-df82-4f00-9eb0-2a1df12fa34a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:57.749982 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:57.749887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"
Sep 11 17:42:58.027325 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:58.027302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts"]
Sep 11 17:42:58.029573 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:42:58.029546    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7d9194_df82_4f00_9eb0_2a1df12fa34a.slice/crio-0136e2b63df0628ac01ef4fe0de62c68196b743e34a930dad924ca647533f08b WatchSource:0}: Error finding container 0136e2b63df0628ac01ef4fe0de62c68196b743e34a930dad924ca647533f08b: Status 404 returned error can't find the container with id 0136e2b63df0628ac01ef4fe0de62c68196b743e34a930dad924ca647533f08b
Sep 11 17:42:58.884934 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:42:58.884898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts" event={"ID":"3b7d9194-df82-4f00-9eb0-2a1df12fa34a","Type":"ContainerStarted","Data":"0136e2b63df0628ac01ef4fe0de62c68196b743e34a930dad924ca647533f08b"}
Sep 11 17:43:00.894747 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.894677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts" event={"ID":"3b7d9194-df82-4f00-9eb0-2a1df12fa34a","Type":"ContainerStarted","Data":"83bdd42b8d3339270e9246169daab895e018f0f830ce8ad500cf5d115963a6d0"}
Sep 11 17:43:00.919430 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.919375    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-45kts" podStartSLOduration=2.079386995 podStartE2EDuration="3.91935836s" totalImagesPullingTime="1.839971365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:00.917347766 +0000 UTC m=+247.680802493" watchObservedRunningTime="2026-09-11 17:43:00.91935836 +0000 UTC m=+247.682813086"
Sep 11 17:43:00.970316 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.970278    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-r2zq7"]
Sep 11 17:43:00.973535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.973501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:00.976583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.976564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fptb6\""
Sep 11 17:43:00.976692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.976606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 17:43:00.976692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.976670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 17:43:00.983082 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:00.983062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-r2zq7"]
Sep 11 17:43:01.057619 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.057576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b7l\" (UniqueName: \"kubernetes.io/projected/0c8d22ad-7337-4f6c-9273-5354eee76ce9-kube-api-access-k5b7l\") pod \"cert-manager-webhook-6486ffd997-r2zq7\" (UID: \"0c8d22ad-7337-4f6c-9273-5354eee76ce9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.057795 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.057628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8d22ad-7337-4f6c-9273-5354eee76ce9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-r2zq7\" (UID: \"0c8d22ad-7337-4f6c-9273-5354eee76ce9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.158466 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.158385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b7l\" (UniqueName: \"kubernetes.io/projected/0c8d22ad-7337-4f6c-9273-5354eee76ce9-kube-api-access-k5b7l\") pod \"cert-manager-webhook-6486ffd997-r2zq7\" (UID: \"0c8d22ad-7337-4f6c-9273-5354eee76ce9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.158466 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.158421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8d22ad-7337-4f6c-9273-5354eee76ce9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-r2zq7\" (UID: \"0c8d22ad-7337-4f6c-9273-5354eee76ce9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.170736 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.170713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c8d22ad-7337-4f6c-9273-5354eee76ce9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-r2zq7\" (UID: \"0c8d22ad-7337-4f6c-9273-5354eee76ce9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.170875 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.170853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b7l\" (UniqueName: \"kubernetes.io/projected/0c8d22ad-7337-4f6c-9273-5354eee76ce9-kube-api-access-k5b7l\") pod \"cert-manager-webhook-6486ffd997-r2zq7\" (UID: \"0c8d22ad-7337-4f6c-9273-5354eee76ce9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.300303 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.300271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:01.580050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.580017    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-r2zq7"]
Sep 11 17:43:01.584167 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:01.584137    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c8d22ad_7337_4f6c_9273_5354eee76ce9.slice/crio-9d831d51c53cb517d42c3182fe33e33ead71c7a044159d26cd6fed9e0c69eeb4 WatchSource:0}: Error finding container 9d831d51c53cb517d42c3182fe33e33ead71c7a044159d26cd6fed9e0c69eeb4: Status 404 returned error can't find the container with id 9d831d51c53cb517d42c3182fe33e33ead71c7a044159d26cd6fed9e0c69eeb4
Sep 11 17:43:01.900622 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.900538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7" event={"ID":"0c8d22ad-7337-4f6c-9273-5354eee76ce9","Type":"ContainerStarted","Data":"9d831d51c53cb517d42c3182fe33e33ead71c7a044159d26cd6fed9e0c69eeb4"}
Sep 11 17:43:01.949147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.949110    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"]
Sep 11 17:43:01.953704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.953682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:01.956895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.956862    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:43:01.956895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.956885    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:43:01.958327 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.958310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z7qc9\""
Sep 11 17:43:01.962993 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:01.962962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"]
Sep 11 17:43:02.065675 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.065635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.065675 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.065677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jnb\" (UniqueName: \"kubernetes.io/projected/1d53227b-7e99-49c1-8153-b69a280df34f-kube-api-access-m4jnb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.065893 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.065742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.166860 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.166769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.166860 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.166826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.166860 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.166852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jnb\" (UniqueName: \"kubernetes.io/projected/1d53227b-7e99-49c1-8153-b69a280df34f-kube-api-access-m4jnb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.167289 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.167133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.167289 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.167216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.179292 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.179263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jnb\" (UniqueName: \"kubernetes.io/projected/1d53227b-7e99-49c1-8153-b69a280df34f-kube-api-access-m4jnb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.265577 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.265537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:02.540764 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.540695    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"]
Sep 11 17:43:02.542870 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:02.542843    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d53227b_7e99_49c1_8153_b69a280df34f.slice/crio-773fcb1c3f916737916962b01f7bdeb4772e130857f0d4bcb48fb6b67586e9b3 WatchSource:0}: Error finding container 773fcb1c3f916737916962b01f7bdeb4772e130857f0d4bcb48fb6b67586e9b3: Status 404 returned error can't find the container with id 773fcb1c3f916737916962b01f7bdeb4772e130857f0d4bcb48fb6b67586e9b3
Sep 11 17:43:02.907130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.907094    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d53227b-7e99-49c1-8153-b69a280df34f" containerID="b7e9e4007f068f9a3bec1bf8c4fe21e293a7ec1a5366690a877e5cb9f0e464c0" exitCode=0
Sep 11 17:43:02.907603 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.907137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx" event={"ID":"1d53227b-7e99-49c1-8153-b69a280df34f","Type":"ContainerDied","Data":"b7e9e4007f068f9a3bec1bf8c4fe21e293a7ec1a5366690a877e5cb9f0e464c0"}
Sep 11 17:43:02.907603 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:02.907180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx" event={"ID":"1d53227b-7e99-49c1-8153-b69a280df34f","Type":"ContainerStarted","Data":"773fcb1c3f916737916962b01f7bdeb4772e130857f0d4bcb48fb6b67586e9b3"}
Sep 11 17:43:04.918970 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:04.918928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7" event={"ID":"0c8d22ad-7337-4f6c-9273-5354eee76ce9","Type":"ContainerStarted","Data":"7a582ef65d4dbc5ae60f5251156a9b6e579f24d5e52eb8a965cd50a9c9296008"}
Sep 11 17:43:04.919408 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:04.919010    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:04.936308 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:04.936247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7" podStartSLOduration=2.088451797 podStartE2EDuration="4.936231266s" totalImagesPullingTime="2.847779469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:04.935480655 +0000 UTC m=+251.698935383" watchObservedRunningTime="2026-09-11 17:43:04.936231266 +0000 UTC m=+251.699685993"
Sep 11 17:43:05.925725 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:05.925694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx" event={"ID":"1d53227b-7e99-49c1-8153-b69a280df34f","Type":"ContainerStarted","Data":"acbdcef723927e0e5bf9a165cc1511a37883e40915e341d4f912c3be902661e0"}
Sep 11 17:43:06.931367 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:06.931331    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d53227b-7e99-49c1-8153-b69a280df34f" containerID="acbdcef723927e0e5bf9a165cc1511a37883e40915e341d4f912c3be902661e0" exitCode=0
Sep 11 17:43:06.931833 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:06.931418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx" event={"ID":"1d53227b-7e99-49c1-8153-b69a280df34f","Type":"ContainerDied","Data":"acbdcef723927e0e5bf9a165cc1511a37883e40915e341d4f912c3be902661e0"}
Sep 11 17:43:07.335456 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.335428    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-78npf"]
Sep 11 17:43:07.338780 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.338760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.341765 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.341747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vxzff\""
Sep 11 17:43:07.347828 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.347805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-78npf"]
Sep 11 17:43:07.408075 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.408039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbj65\" (UniqueName: \"kubernetes.io/projected/910bef48-2d74-4744-879c-6067e493ae6a-kube-api-access-mbj65\") pod \"cert-manager-cainjector-559cdd8c87-78npf\" (UID: \"910bef48-2d74-4744-879c-6067e493ae6a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.408227 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.408098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/910bef48-2d74-4744-879c-6067e493ae6a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-78npf\" (UID: \"910bef48-2d74-4744-879c-6067e493ae6a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.508962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.508928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/910bef48-2d74-4744-879c-6067e493ae6a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-78npf\" (UID: \"910bef48-2d74-4744-879c-6067e493ae6a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.509127 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.508978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbj65\" (UniqueName: \"kubernetes.io/projected/910bef48-2d74-4744-879c-6067e493ae6a-kube-api-access-mbj65\") pod \"cert-manager-cainjector-559cdd8c87-78npf\" (UID: \"910bef48-2d74-4744-879c-6067e493ae6a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.517350 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.517315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/910bef48-2d74-4744-879c-6067e493ae6a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-78npf\" (UID: \"910bef48-2d74-4744-879c-6067e493ae6a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.517475 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.517375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbj65\" (UniqueName: \"kubernetes.io/projected/910bef48-2d74-4744-879c-6067e493ae6a-kube-api-access-mbj65\") pod \"cert-manager-cainjector-559cdd8c87-78npf\" (UID: \"910bef48-2d74-4744-879c-6067e493ae6a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.647935 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.647825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf"
Sep 11 17:43:07.916730 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.916706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-78npf"]
Sep 11 17:43:07.918461 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:07.918432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod910bef48_2d74_4744_879c_6067e493ae6a.slice/crio-5e63395e3a9e2a3b842cf89334d468cc796b080fcb5e2428c26fc0cef02b1ad5 WatchSource:0}: Error finding container 5e63395e3a9e2a3b842cf89334d468cc796b080fcb5e2428c26fc0cef02b1ad5: Status 404 returned error can't find the container with id 5e63395e3a9e2a3b842cf89334d468cc796b080fcb5e2428c26fc0cef02b1ad5
Sep 11 17:43:07.940705 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.940676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf" event={"ID":"910bef48-2d74-4744-879c-6067e493ae6a","Type":"ContainerStarted","Data":"5e63395e3a9e2a3b842cf89334d468cc796b080fcb5e2428c26fc0cef02b1ad5"}
Sep 11 17:43:07.943093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.943068    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d53227b-7e99-49c1-8153-b69a280df34f" containerID="8cb67218d1a2267a1fa77273a6c4a46d45809940dc7f0fe4d1afb5f54b0d37e9" exitCode=0
Sep 11 17:43:07.943188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:07.943104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx" event={"ID":"1d53227b-7e99-49c1-8153-b69a280df34f","Type":"ContainerDied","Data":"8cb67218d1a2267a1fa77273a6c4a46d45809940dc7f0fe4d1afb5f54b0d37e9"}
Sep 11 17:43:08.948428 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:08.948389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf" event={"ID":"910bef48-2d74-4744-879c-6067e493ae6a","Type":"ContainerStarted","Data":"c277dc6c29fb1b6c004b12160c3ce40f79e72215b4ff278df6f91374a88a451a"}
Sep 11 17:43:08.965682 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:08.965601    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-78npf" podStartSLOduration=1.9655774940000001 podStartE2EDuration="1.965577494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:08.965293059 +0000 UTC m=+255.728747787" watchObservedRunningTime="2026-09-11 17:43:08.965577494 +0000 UTC m=+255.729032221"
Sep 11 17:43:09.076849 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.076826    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:09.122065 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.122038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4jnb\" (UniqueName: \"kubernetes.io/projected/1d53227b-7e99-49c1-8153-b69a280df34f-kube-api-access-m4jnb\") pod \"1d53227b-7e99-49c1-8153-b69a280df34f\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") "
Sep 11 17:43:09.122223 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.122073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-util\") pod \"1d53227b-7e99-49c1-8153-b69a280df34f\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") "
Sep 11 17:43:09.122223 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.122098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-bundle\") pod \"1d53227b-7e99-49c1-8153-b69a280df34f\" (UID: \"1d53227b-7e99-49c1-8153-b69a280df34f\") "
Sep 11 17:43:09.122511 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.122459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-bundle" (OuterVolumeSpecName: "bundle") pod "1d53227b-7e99-49c1-8153-b69a280df34f" (UID: "1d53227b-7e99-49c1-8153-b69a280df34f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:09.124286 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.124257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d53227b-7e99-49c1-8153-b69a280df34f-kube-api-access-m4jnb" (OuterVolumeSpecName: "kube-api-access-m4jnb") pod "1d53227b-7e99-49c1-8153-b69a280df34f" (UID: "1d53227b-7e99-49c1-8153-b69a280df34f"). InnerVolumeSpecName "kube-api-access-m4jnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:09.126685 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.126652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-util" (OuterVolumeSpecName: "util") pod "1d53227b-7e99-49c1-8153-b69a280df34f" (UID: "1d53227b-7e99-49c1-8153-b69a280df34f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:09.228220 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.223847    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4jnb\" (UniqueName: \"kubernetes.io/projected/1d53227b-7e99-49c1-8153-b69a280df34f-kube-api-access-m4jnb\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:09.228220 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.223903    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:09.228220 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.223931    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d53227b-7e99-49c1-8153-b69a280df34f-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:09.954577 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.954545    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx"
Sep 11 17:43:09.954577 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.954561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdwssx" event={"ID":"1d53227b-7e99-49c1-8153-b69a280df34f","Type":"ContainerDied","Data":"773fcb1c3f916737916962b01f7bdeb4772e130857f0d4bcb48fb6b67586e9b3"}
Sep 11 17:43:09.955097 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:09.954608    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773fcb1c3f916737916962b01f7bdeb4772e130857f0d4bcb48fb6b67586e9b3"
Sep 11 17:43:10.928165 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:10.928134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-r2zq7"
Sep 11 17:43:21.499143 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499068    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lzbfz"]
Sep 11 17:43:21.499531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="util"
Sep 11 17:43:21.499531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499498    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="util"
Sep 11 17:43:21.499531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="pull"
Sep 11 17:43:21.499531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="pull"
Sep 11 17:43:21.499531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="extract"
Sep 11 17:43:21.499531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="extract"
Sep 11 17:43:21.499716 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.499580    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d53227b-7e99-49c1-8153-b69a280df34f" containerName="extract"
Sep 11 17:43:21.552324 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.552287    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lzbfz"]
Sep 11 17:43:21.552516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.552390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.555518 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.555483    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s6rw6\""
Sep 11 17:43:21.628865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.628824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb92r\" (UniqueName: \"kubernetes.io/projected/0a477626-fdf7-42d9-8266-85790bc2a603-kube-api-access-pb92r\") pod \"cert-manager-55b4d954d5-lzbfz\" (UID: \"0a477626-fdf7-42d9-8266-85790bc2a603\") " pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.629045 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.628941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a477626-fdf7-42d9-8266-85790bc2a603-bound-sa-token\") pod \"cert-manager-55b4d954d5-lzbfz\" (UID: \"0a477626-fdf7-42d9-8266-85790bc2a603\") " pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.730188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.730155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a477626-fdf7-42d9-8266-85790bc2a603-bound-sa-token\") pod \"cert-manager-55b4d954d5-lzbfz\" (UID: \"0a477626-fdf7-42d9-8266-85790bc2a603\") " pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.730350 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.730196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb92r\" (UniqueName: \"kubernetes.io/projected/0a477626-fdf7-42d9-8266-85790bc2a603-kube-api-access-pb92r\") pod \"cert-manager-55b4d954d5-lzbfz\" (UID: \"0a477626-fdf7-42d9-8266-85790bc2a603\") " pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.738821 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.738792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a477626-fdf7-42d9-8266-85790bc2a603-bound-sa-token\") pod \"cert-manager-55b4d954d5-lzbfz\" (UID: \"0a477626-fdf7-42d9-8266-85790bc2a603\") " pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.738930 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.738836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb92r\" (UniqueName: \"kubernetes.io/projected/0a477626-fdf7-42d9-8266-85790bc2a603-kube-api-access-pb92r\") pod \"cert-manager-55b4d954d5-lzbfz\" (UID: \"0a477626-fdf7-42d9-8266-85790bc2a603\") " pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:21.861231 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:21.861202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lzbfz"
Sep 11 17:43:22.137144 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:22.137120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lzbfz"]
Sep 11 17:43:22.139478 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:22.139449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a477626_fdf7_42d9_8266_85790bc2a603.slice/crio-2caa4d0b007310388361e4ad3c88c5825dbcb1d5185a3fa9d8cac000d1510052 WatchSource:0}: Error finding container 2caa4d0b007310388361e4ad3c88c5825dbcb1d5185a3fa9d8cac000d1510052: Status 404 returned error can't find the container with id 2caa4d0b007310388361e4ad3c88c5825dbcb1d5185a3fa9d8cac000d1510052
Sep 11 17:43:23.012005 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:23.011965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lzbfz" event={"ID":"0a477626-fdf7-42d9-8266-85790bc2a603","Type":"ContainerStarted","Data":"0ae09cb8873597e9d65a4687cf73324ee843c505d70d686fb7c1dcd0eeaec80c"}
Sep 11 17:43:23.012005 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:23.012011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lzbfz" event={"ID":"0a477626-fdf7-42d9-8266-85790bc2a603","Type":"ContainerStarted","Data":"2caa4d0b007310388361e4ad3c88c5825dbcb1d5185a3fa9d8cac000d1510052"}
Sep 11 17:43:23.028583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:23.028532    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lzbfz" podStartSLOduration=2.028516805 podStartE2EDuration="2.028516805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:23.027353916 +0000 UTC m=+269.790808642" watchObservedRunningTime="2026-09-11 17:43:23.028516805 +0000 UTC m=+269.791971526"
Sep 11 17:43:27.503060 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.503026    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"]
Sep 11 17:43:27.509570 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.509541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.512646 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.512621    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:43:27.514060 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.514038    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:43:27.514162 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.514047    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z7qc9\""
Sep 11 17:43:27.516220 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.516197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"]
Sep 11 17:43:27.580811 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.580774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.580811 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.580817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.581066 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.580978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9wl\" (UniqueName: \"kubernetes.io/projected/aa468c6f-da80-4a4a-b670-fecf5a04dd44-kube-api-access-lt9wl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.682242 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.682205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9wl\" (UniqueName: \"kubernetes.io/projected/aa468c6f-da80-4a4a-b670-fecf5a04dd44-kube-api-access-lt9wl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.682242 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.682243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.682531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.682267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.682636 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.682618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.682712 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.682688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.690788 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.690764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9wl\" (UniqueName: \"kubernetes.io/projected/aa468c6f-da80-4a4a-b670-fecf5a04dd44-kube-api-access-lt9wl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:27.820041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:27.820017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:28.090810 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:28.090783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"]
Sep 11 17:43:28.093578 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:28.093461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa468c6f_da80_4a4a_b670_fecf5a04dd44.slice/crio-bbdfcbdc2e26bfcdb0775c1dbc7ca0f3a9e00850f5336e479f9686f14d4a9c36 WatchSource:0}: Error finding container bbdfcbdc2e26bfcdb0775c1dbc7ca0f3a9e00850f5336e479f9686f14d4a9c36: Status 404 returned error can't find the container with id bbdfcbdc2e26bfcdb0775c1dbc7ca0f3a9e00850f5336e479f9686f14d4a9c36
Sep 11 17:43:29.039723 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:29.039632    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerID="6cb57189c3e4e5617bcabe23972900b4887e0939478a5f12ce074f69896ccbbf" exitCode=0
Sep 11 17:43:29.040133 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:29.039726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6" event={"ID":"aa468c6f-da80-4a4a-b670-fecf5a04dd44","Type":"ContainerDied","Data":"6cb57189c3e4e5617bcabe23972900b4887e0939478a5f12ce074f69896ccbbf"}
Sep 11 17:43:29.040133 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:29.039773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6" event={"ID":"aa468c6f-da80-4a4a-b670-fecf5a04dd44","Type":"ContainerStarted","Data":"bbdfcbdc2e26bfcdb0775c1dbc7ca0f3a9e00850f5336e479f9686f14d4a9c36"}
Sep 11 17:43:30.046748 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:30.046667    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerID="35295d0785359e4efb663808df8233323c75097e08ccc0abc331daa927abe62f" exitCode=0
Sep 11 17:43:30.046748 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:30.046715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6" event={"ID":"aa468c6f-da80-4a4a-b670-fecf5a04dd44","Type":"ContainerDied","Data":"35295d0785359e4efb663808df8233323c75097e08ccc0abc331daa927abe62f"}
Sep 11 17:43:31.053630 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:31.053594    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerID="46a7fcb964c9a27ca217a6f38b4d183920bb9fe960e38ea9d0640801bb979018" exitCode=0
Sep 11 17:43:31.054059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:31.053648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6" event={"ID":"aa468c6f-da80-4a4a-b670-fecf5a04dd44","Type":"ContainerDied","Data":"46a7fcb964c9a27ca217a6f38b4d183920bb9fe960e38ea9d0640801bb979018"}
Sep 11 17:43:32.190038 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.190010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:32.318938 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.318911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-bundle\") pod \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") "
Sep 11 17:43:32.319184 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.318950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt9wl\" (UniqueName: \"kubernetes.io/projected/aa468c6f-da80-4a4a-b670-fecf5a04dd44-kube-api-access-lt9wl\") pod \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") "
Sep 11 17:43:32.319184 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.319012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-util\") pod \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\" (UID: \"aa468c6f-da80-4a4a-b670-fecf5a04dd44\") "
Sep 11 17:43:32.319799 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.319772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-bundle" (OuterVolumeSpecName: "bundle") pod "aa468c6f-da80-4a4a-b670-fecf5a04dd44" (UID: "aa468c6f-da80-4a4a-b670-fecf5a04dd44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:32.321237 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.321210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa468c6f-da80-4a4a-b670-fecf5a04dd44-kube-api-access-lt9wl" (OuterVolumeSpecName: "kube-api-access-lt9wl") pod "aa468c6f-da80-4a4a-b670-fecf5a04dd44" (UID: "aa468c6f-da80-4a4a-b670-fecf5a04dd44"). InnerVolumeSpecName "kube-api-access-lt9wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:32.325027 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.325000    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-util" (OuterVolumeSpecName: "util") pod "aa468c6f-da80-4a4a-b670-fecf5a04dd44" (UID: "aa468c6f-da80-4a4a-b670-fecf5a04dd44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:32.420396 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.420359    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:32.420396 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.420389    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9wl\" (UniqueName: \"kubernetes.io/projected/aa468c6f-da80-4a4a-b670-fecf5a04dd44-kube-api-access-lt9wl\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:32.420396 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:32.420399    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa468c6f-da80-4a4a-b670-fecf5a04dd44-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:33.071137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:33.071105    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6"
Sep 11 17:43:33.071315 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:33.071105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9dr6" event={"ID":"aa468c6f-da80-4a4a-b670-fecf5a04dd44","Type":"ContainerDied","Data":"bbdfcbdc2e26bfcdb0775c1dbc7ca0f3a9e00850f5336e479f9686f14d4a9c36"}
Sep 11 17:43:33.071315 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:33.071222    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbdfcbdc2e26bfcdb0775c1dbc7ca0f3a9e00850f5336e479f9686f14d4a9c36"
Sep 11 17:43:38.313615 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.313572    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"]
Sep 11 17:43:38.314092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.313993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="util"
Sep 11 17:43:38.314092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.314010    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="util"
Sep 11 17:43:38.314092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.314025    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="extract"
Sep 11 17:43:38.314092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.314033    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="extract"
Sep 11 17:43:38.314092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.314052    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="pull"
Sep 11 17:43:38.314092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.314059    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="pull"
Sep 11 17:43:38.314404 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.314138    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa468c6f-da80-4a4a-b670-fecf5a04dd44" containerName="extract"
Sep 11 17:43:38.318544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.318522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.321672 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.321651    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:43:38.321773 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.321733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:43:38.322892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.322876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z7qc9\""
Sep 11 17:43:38.326877 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.326854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"]
Sep 11 17:43:38.474077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.474045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxlx\" (UniqueName: \"kubernetes.io/projected/6f326c17-9b44-4373-9336-5d5340a9ecd3-kube-api-access-tbxlx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.474077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.474082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.474306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.474106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.575544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.575481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxlx\" (UniqueName: \"kubernetes.io/projected/6f326c17-9b44-4373-9336-5d5340a9ecd3-kube-api-access-tbxlx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.575708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.575554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.575708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.575583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.575953 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.575937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.576032 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.575978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.588160 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.588127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxlx\" (UniqueName: \"kubernetes.io/projected/6f326c17-9b44-4373-9336-5d5340a9ecd3-kube-api-access-tbxlx\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.628165 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.628122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:38.901150 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:38.901090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"]
Sep 11 17:43:38.903558 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:38.903531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f326c17_9b44_4373_9336_5d5340a9ecd3.slice/crio-8a8527186dcf4d50b184fff162885445614133a352a1b3afe617c3a85ca57a16 WatchSource:0}: Error finding container 8a8527186dcf4d50b184fff162885445614133a352a1b3afe617c3a85ca57a16: Status 404 returned error can't find the container with id 8a8527186dcf4d50b184fff162885445614133a352a1b3afe617c3a85ca57a16
Sep 11 17:43:39.102376 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:39.102342    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerID="a7ed56a69867bb2cf2209839c1c5c8ff17d157aca4e93d6b48ac68cfbb1f885b" exitCode=0
Sep 11 17:43:39.102571 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:39.102429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9" event={"ID":"6f326c17-9b44-4373-9336-5d5340a9ecd3","Type":"ContainerDied","Data":"a7ed56a69867bb2cf2209839c1c5c8ff17d157aca4e93d6b48ac68cfbb1f885b"}
Sep 11 17:43:39.102571 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:39.102479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9" event={"ID":"6f326c17-9b44-4373-9336-5d5340a9ecd3","Type":"ContainerStarted","Data":"8a8527186dcf4d50b184fff162885445614133a352a1b3afe617c3a85ca57a16"}
Sep 11 17:43:40.109050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.109021    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerID="29a116228c61476ee4813ab7d0ac2bb9eeed024b2a83f9c881a321e7f098eb63" exitCode=0
Sep 11 17:43:40.109531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.109094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9" event={"ID":"6f326c17-9b44-4373-9336-5d5340a9ecd3","Type":"ContainerDied","Data":"29a116228c61476ee4813ab7d0ac2bb9eeed024b2a83f9c881a321e7f098eb63"}
Sep 11 17:43:40.662210 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.662175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"]
Sep 11 17:43:40.665397 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.665382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.668839 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.668819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 17:43:40.669475 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.669460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 17:43:40.669874 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.669846    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 17:43:40.669874 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.669873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jvzqs\""
Sep 11 17:43:40.670017 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.669875    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 17:43:40.694876 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.694852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"]
Sep 11 17:43:40.796383 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.796345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jbx\" (UniqueName: \"kubernetes.io/projected/47ec1ed1-e924-4007-9a80-f42ccf6c9350-kube-api-access-l6jbx\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.796584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.796389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47ec1ed1-e924-4007-9a80-f42ccf6c9350-tmp\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.796584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.796509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47ec1ed1-e924-4007-9a80-f42ccf6c9350-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.796584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.796560    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47ec1ed1-e924-4007-9a80-f42ccf6c9350-manifests\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.796694 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.796618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47ec1ed1-e924-4007-9a80-f42ccf6c9350-webhook-cert\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.897185 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.897149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jbx\" (UniqueName: \"kubernetes.io/projected/47ec1ed1-e924-4007-9a80-f42ccf6c9350-kube-api-access-l6jbx\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.897185 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.897191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47ec1ed1-e924-4007-9a80-f42ccf6c9350-tmp\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.897441 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.897229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47ec1ed1-e924-4007-9a80-f42ccf6c9350-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.897441 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.897281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47ec1ed1-e924-4007-9a80-f42ccf6c9350-manifests\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.897441 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.897309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47ec1ed1-e924-4007-9a80-f42ccf6c9350-webhook-cert\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.897717 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.897689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/47ec1ed1-e924-4007-9a80-f42ccf6c9350-manifests\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.899766 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.899743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47ec1ed1-e924-4007-9a80-f42ccf6c9350-tmp\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.899893 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.899833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/47ec1ed1-e924-4007-9a80-f42ccf6c9350-webhook-cert\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.899981 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.899961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/47ec1ed1-e924-4007-9a80-f42ccf6c9350-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.906734 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.906707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jbx\" (UniqueName: \"kubernetes.io/projected/47ec1ed1-e924-4007-9a80-f42ccf6c9350-kube-api-access-l6jbx\") pod \"opendatahub-operator-controller-manager-86997d8dbf-fkzwm\" (UID: \"47ec1ed1-e924-4007-9a80-f42ccf6c9350\") " pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:40.975704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:40.975601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:41.116204 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:41.116165    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerID="d03aa402749c3c17fa9feae54681a2a660fdec2a990839ecc033f311d99ae2c8" exitCode=0
Sep 11 17:43:41.116611 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:41.116207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9" event={"ID":"6f326c17-9b44-4373-9336-5d5340a9ecd3","Type":"ContainerDied","Data":"d03aa402749c3c17fa9feae54681a2a660fdec2a990839ecc033f311d99ae2c8"}
Sep 11 17:43:41.263431 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:41.263392    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"]
Sep 11 17:43:41.265899 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:43:41.265866    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ec1ed1_e924_4007_9a80_f42ccf6c9350.slice/crio-63331d7e808277d496321937b784c2863c89fa90b92d70a26179527a158dd70d WatchSource:0}: Error finding container 63331d7e808277d496321937b784c2863c89fa90b92d70a26179527a158dd70d: Status 404 returned error can't find the container with id 63331d7e808277d496321937b784c2863c89fa90b92d70a26179527a158dd70d
Sep 11 17:43:42.124784 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.124744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm" event={"ID":"47ec1ed1-e924-4007-9a80-f42ccf6c9350","Type":"ContainerStarted","Data":"63331d7e808277d496321937b784c2863c89fa90b92d70a26179527a158dd70d"}
Sep 11 17:43:42.280412 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.280380    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:42.312713 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.312683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbxlx\" (UniqueName: \"kubernetes.io/projected/6f326c17-9b44-4373-9336-5d5340a9ecd3-kube-api-access-tbxlx\") pod \"6f326c17-9b44-4373-9336-5d5340a9ecd3\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") "
Sep 11 17:43:42.312892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.312754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-util\") pod \"6f326c17-9b44-4373-9336-5d5340a9ecd3\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") "
Sep 11 17:43:42.312892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.312825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-bundle\") pod \"6f326c17-9b44-4373-9336-5d5340a9ecd3\" (UID: \"6f326c17-9b44-4373-9336-5d5340a9ecd3\") "
Sep 11 17:43:42.313935 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.313903    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-bundle" (OuterVolumeSpecName: "bundle") pod "6f326c17-9b44-4373-9336-5d5340a9ecd3" (UID: "6f326c17-9b44-4373-9336-5d5340a9ecd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:42.315983 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.315944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f326c17-9b44-4373-9336-5d5340a9ecd3-kube-api-access-tbxlx" (OuterVolumeSpecName: "kube-api-access-tbxlx") pod "6f326c17-9b44-4373-9336-5d5340a9ecd3" (UID: "6f326c17-9b44-4373-9336-5d5340a9ecd3"). InnerVolumeSpecName "kube-api-access-tbxlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:42.322115 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.322085    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-util" (OuterVolumeSpecName: "util") pod "6f326c17-9b44-4373-9336-5d5340a9ecd3" (UID: "6f326c17-9b44-4373-9336-5d5340a9ecd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:42.414413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.414317    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:42.414413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.414357    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxlx\" (UniqueName: \"kubernetes.io/projected/6f326c17-9b44-4373-9336-5d5340a9ecd3-kube-api-access-tbxlx\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:42.414413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:42.414371    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f326c17-9b44-4373-9336-5d5340a9ecd3-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:43:43.132174 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:43.132142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9" event={"ID":"6f326c17-9b44-4373-9336-5d5340a9ecd3","Type":"ContainerDied","Data":"8a8527186dcf4d50b184fff162885445614133a352a1b3afe617c3a85ca57a16"}
Sep 11 17:43:43.132600 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:43.132191    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rhts9"
Sep 11 17:43:43.132600 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:43.132195    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a8527186dcf4d50b184fff162885445614133a352a1b3afe617c3a85ca57a16"
Sep 11 17:43:45.143918 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:45.143883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm" event={"ID":"47ec1ed1-e924-4007-9a80-f42ccf6c9350","Type":"ContainerStarted","Data":"7c01449997c666b4725e1532cf98ebe8926052e690e56b1a9a9e99d4957fdf4b"}
Sep 11 17:43:46.150472 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:46.150436    2563 generic.go:356] "Generic (PLEG): container finished" podID="47ec1ed1-e924-4007-9a80-f42ccf6c9350" containerID="7c01449997c666b4725e1532cf98ebe8926052e690e56b1a9a9e99d4957fdf4b" exitCode=0
Sep 11 17:43:46.150928 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:46.150528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm" event={"ID":"47ec1ed1-e924-4007-9a80-f42ccf6c9350","Type":"ContainerDied","Data":"7c01449997c666b4725e1532cf98ebe8926052e690e56b1a9a9e99d4957fdf4b"}
Sep 11 17:43:47.161087 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:47.160995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm" event={"ID":"47ec1ed1-e924-4007-9a80-f42ccf6c9350","Type":"ContainerStarted","Data":"ccad75ce431ef77e18b74617b36cabf745c3b24c5b8ecc15d71c783e24c20d6b"}
Sep 11 17:43:47.161534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:47.161095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:47.188565 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:47.188514    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm" podStartSLOduration=2.505260083 podStartE2EDuration="7.188477801s" totalImagesPullingTime="3.693439738s" totalInitContainerRuntime="989.77798ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:47.187307101 +0000 UTC m=+293.950761828" watchObservedRunningTime="2026-09-11 17:43:47.188477801 +0000 UTC m=+293.951932604"
Sep 11 17:43:53.662972 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:53.662941    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:43:53.665157 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:53.665137    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:43:53.670231 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:53.670210    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:43:58.167647 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:58.167615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86997d8dbf-fkzwm"
Sep 11 17:43:59.603137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603098    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"]
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603389    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="pull"
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603399    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="pull"
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603413    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="extract"
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="extract"
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603427    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="util"
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603432    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="util"
Sep 11 17:43:59.603534 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.603479    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f326c17-9b44-4373-9336-5d5340a9ecd3" containerName="extract"
Sep 11 17:43:59.609054 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.609037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.614247 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.614220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 17:43:59.614895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.614878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:43:59.615924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.615903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 17:43:59.616115 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.616095    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 17:43:59.616201 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.616184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kkfd2\""
Sep 11 17:43:59.616443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.616425    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 17:43:59.657070 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.657029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac8da5ee-1613-499f-8759-9d769c40b605-metrics-cert\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.657251 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.657074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac8da5ee-1613-499f-8759-9d769c40b605-manager-config\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.657251 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.657145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4mt\" (UniqueName: \"kubernetes.io/projected/ac8da5ee-1613-499f-8759-9d769c40b605-kube-api-access-nm4mt\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.657251 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.657180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac8da5ee-1613-499f-8759-9d769c40b605-cert\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.705718 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.705689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"]
Sep 11 17:43:59.758031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.757995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac8da5ee-1613-499f-8759-9d769c40b605-metrics-cert\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.758249 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.758054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac8da5ee-1613-499f-8759-9d769c40b605-manager-config\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.758249 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.758117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4mt\" (UniqueName: \"kubernetes.io/projected/ac8da5ee-1613-499f-8759-9d769c40b605-kube-api-access-nm4mt\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.758249 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.758158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac8da5ee-1613-499f-8759-9d769c40b605-cert\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.759090 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.759059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac8da5ee-1613-499f-8759-9d769c40b605-manager-config\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.761284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.761259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac8da5ee-1613-499f-8759-9d769c40b605-cert\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.761402 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.761355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac8da5ee-1613-499f-8759-9d769c40b605-metrics-cert\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.780180 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.780151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4mt\" (UniqueName: \"kubernetes.io/projected/ac8da5ee-1613-499f-8759-9d769c40b605-kube-api-access-nm4mt\") pod \"lws-controller-manager-6476dc957d-595x9\" (UID: \"ac8da5ee-1613-499f-8759-9d769c40b605\") " pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:43:59.918654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:43:59.918572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:44:00.230969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:00.230942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"]
Sep 11 17:44:00.232457 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:44:00.232431    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8da5ee_1613_499f_8759_9d769c40b605.slice/crio-02d02d076f6c34bb1038a9f36054bc85fa5e0518ce49881e9da6b924c94fb9c6 WatchSource:0}: Error finding container 02d02d076f6c34bb1038a9f36054bc85fa5e0518ce49881e9da6b924c94fb9c6: Status 404 returned error can't find the container with id 02d02d076f6c34bb1038a9f36054bc85fa5e0518ce49881e9da6b924c94fb9c6
Sep 11 17:44:00.234319 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:00.234302    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:44:01.227098 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:01.227059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9" event={"ID":"ac8da5ee-1613-499f-8759-9d769c40b605","Type":"ContainerStarted","Data":"02d02d076f6c34bb1038a9f36054bc85fa5e0518ce49881e9da6b924c94fb9c6"}
Sep 11 17:44:02.234085 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:02.234049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9" event={"ID":"ac8da5ee-1613-499f-8759-9d769c40b605","Type":"ContainerStarted","Data":"933baa02da5f469cc22eb7cc4644259cf116ae516fdb98eb6dd191ceaa32a5e3"}
Sep 11 17:44:02.234510 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:02.234177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:44:02.253603 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:02.253553    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9" podStartSLOduration=1.766198888 podStartE2EDuration="3.253536753s" totalImagesPullingTime="1.487337865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:02.252400278 +0000 UTC m=+309.015855005" watchObservedRunningTime="2026-09-11 17:44:02.253536753 +0000 UTC m=+309.016991480"
Sep 11 17:44:11.175241 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.175207    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qsqsw"]
Sep 11 17:44:11.177502 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.177469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.180648 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.180626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-49sw8\""
Sep 11 17:44:11.180802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.180780    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 17:44:11.196107 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.196078    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qsqsw"]
Sep 11 17:44:11.255428 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.255380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsq4f\" (UniqueName: \"kubernetes.io/projected/f2067629-41a1-4d38-8b75-5f232d8b8925-kube-api-access-hsq4f\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.255628 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.255539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2067629-41a1-4d38-8b75-5f232d8b8925-cert\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.355976 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.355948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2067629-41a1-4d38-8b75-5f232d8b8925-cert\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.356169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.356011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsq4f\" (UniqueName: \"kubernetes.io/projected/f2067629-41a1-4d38-8b75-5f232d8b8925-kube-api-access-hsq4f\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.356169 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:44:11.356112    2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 17:44:11.356289 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:44:11.356176    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2067629-41a1-4d38-8b75-5f232d8b8925-cert podName:f2067629-41a1-4d38-8b75-5f232d8b8925 nodeName:}" failed. No retries permitted until 2026-09-11 17:44:11.856161606 +0000 UTC m=+318.619616311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f2067629-41a1-4d38-8b75-5f232d8b8925-cert") pod "odh-model-controller-7998c67dc4-qsqsw" (UID: "f2067629-41a1-4d38-8b75-5f232d8b8925") : secret "odh-model-controller-webhook-cert" not found
Sep 11 17:44:11.369777 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.369749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsq4f\" (UniqueName: \"kubernetes.io/projected/f2067629-41a1-4d38-8b75-5f232d8b8925-kube-api-access-hsq4f\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.860299 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.860262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2067629-41a1-4d38-8b75-5f232d8b8925-cert\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.862902 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.862879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2067629-41a1-4d38-8b75-5f232d8b8925-cert\") pod \"odh-model-controller-7998c67dc4-qsqsw\" (UID: \"f2067629-41a1-4d38-8b75-5f232d8b8925\") " pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:11.982315 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.982280    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vz62r"]
Sep 11 17:44:11.984569 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.984550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:11.996205 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:11.996186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 17:44:12.014277 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.014243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vz62r"]
Sep 11 17:44:12.061745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.061715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f59910-f0ba-4d2d-b532-ffd0af390f9f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz62r\" (UID: \"d4f59910-f0ba-4d2d-b532-ffd0af390f9f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.061745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.061749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8q8\" (UniqueName: \"kubernetes.io/projected/d4f59910-f0ba-4d2d-b532-ffd0af390f9f-kube-api-access-2p8q8\") pod \"model-serving-api-54d89bd5fc-vz62r\" (UID: \"d4f59910-f0ba-4d2d-b532-ffd0af390f9f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.088682 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.088649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:12.162724 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.162695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f59910-f0ba-4d2d-b532-ffd0af390f9f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz62r\" (UID: \"d4f59910-f0ba-4d2d-b532-ffd0af390f9f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.162724 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.162729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8q8\" (UniqueName: \"kubernetes.io/projected/d4f59910-f0ba-4d2d-b532-ffd0af390f9f-kube-api-access-2p8q8\") pod \"model-serving-api-54d89bd5fc-vz62r\" (UID: \"d4f59910-f0ba-4d2d-b532-ffd0af390f9f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.165222 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.165196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4f59910-f0ba-4d2d-b532-ffd0af390f9f-tls-certs\") pod \"model-serving-api-54d89bd5fc-vz62r\" (UID: \"d4f59910-f0ba-4d2d-b532-ffd0af390f9f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.187108 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.187072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8q8\" (UniqueName: \"kubernetes.io/projected/d4f59910-f0ba-4d2d-b532-ffd0af390f9f-kube-api-access-2p8q8\") pod \"model-serving-api-54d89bd5fc-vz62r\" (UID: \"d4f59910-f0ba-4d2d-b532-ffd0af390f9f\") " pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.294946 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.294915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:12.379110 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.379043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-qsqsw"]
Sep 11 17:44:12.381253 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:44:12.381226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2067629_41a1_4d38_8b75_5f232d8b8925.slice/crio-d72931c8ca13183e443a9857b378468282783042c53dd9d862d1ba412d181747 WatchSource:0}: Error finding container d72931c8ca13183e443a9857b378468282783042c53dd9d862d1ba412d181747: Status 404 returned error can't find the container with id d72931c8ca13183e443a9857b378468282783042c53dd9d862d1ba412d181747
Sep 11 17:44:12.572228 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.572201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vz62r"]
Sep 11 17:44:12.574317 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:44:12.574277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f59910_f0ba_4d2d_b532_ffd0af390f9f.slice/crio-b56896a812e1830190d7f924151b91b264755ee05abb1d752ccd6cc56a930982 WatchSource:0}: Error finding container b56896a812e1830190d7f924151b91b264755ee05abb1d752ccd6cc56a930982: Status 404 returned error can't find the container with id b56896a812e1830190d7f924151b91b264755ee05abb1d752ccd6cc56a930982
Sep 11 17:44:12.987131 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.987101    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"]
Sep 11 17:44:12.989201 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.989183    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:12.999835 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:12.999757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:44:13.000182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.000056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 17:44:13.000182 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.000056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 17:44:13.000365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.000191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9zrvc\""
Sep 11 17:44:13.020058 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.019978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"]
Sep 11 17:44:13.070956 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.070910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.070956 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.070959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071174 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.070992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071174 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.071028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071174 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.071050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071174 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.071132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.071197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.071224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxc4\" (UniqueName: \"kubernetes.io/projected/4b9f65a6-44d2-4bf1-8711-6939def63e7e-kube-api-access-wqxc4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.071360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.071267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172454 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172658 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172658 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172658 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172658 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.172873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxc4\" (UniqueName: \"kubernetes.io/projected/4b9f65a6-44d2-4bf1-8711-6939def63e7e-kube-api-access-wqxc4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.173080 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.172915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.173080 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.173050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.173345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.173300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.173345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.173321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.173621 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.173601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.175744 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.175718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.175865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.175848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.183595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.183567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9f65a6-44d2-4bf1-8711-6939def63e7e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.184870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.184847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxc4\" (UniqueName: \"kubernetes.io/projected/4b9f65a6-44d2-4bf1-8711-6939def63e7e-kube-api-access-wqxc4\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh\" (UID: \"4b9f65a6-44d2-4bf1-8711-6939def63e7e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.245335 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.245183    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6476dc957d-595x9"
Sep 11 17:44:13.297419 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.297377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r" event={"ID":"d4f59910-f0ba-4d2d-b532-ffd0af390f9f","Type":"ContainerStarted","Data":"b56896a812e1830190d7f924151b91b264755ee05abb1d752ccd6cc56a930982"}
Sep 11 17:44:13.303083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.302844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:13.303083 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.302944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw" event={"ID":"f2067629-41a1-4d38-8b75-5f232d8b8925","Type":"ContainerStarted","Data":"d72931c8ca13183e443a9857b378468282783042c53dd9d862d1ba412d181747"}
Sep 11 17:44:13.614907 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:13.614229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"]
Sep 11 17:44:13.623049 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:44:13.623010    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9f65a6_44d2_4bf1_8711_6939def63e7e.slice/crio-034fc1f955436fd1fdc85422bbfac7a9485eae0a3f8ef7d3ddd89b36cc5836b4 WatchSource:0}: Error finding container 034fc1f955436fd1fdc85422bbfac7a9485eae0a3f8ef7d3ddd89b36cc5836b4: Status 404 returned error can't find the container with id 034fc1f955436fd1fdc85422bbfac7a9485eae0a3f8ef7d3ddd89b36cc5836b4
Sep 11 17:44:14.312117 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:14.312082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh" event={"ID":"4b9f65a6-44d2-4bf1-8711-6939def63e7e","Type":"ContainerStarted","Data":"034fc1f955436fd1fdc85422bbfac7a9485eae0a3f8ef7d3ddd89b36cc5836b4"}
Sep 11 17:44:17.333980 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.333945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r" event={"ID":"d4f59910-f0ba-4d2d-b532-ffd0af390f9f","Type":"ContainerStarted","Data":"df2013cf059ba2f85bd9495776b1b54fcef31a158245318a97920fd4a114d27e"}
Sep 11 17:44:17.334453 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.334067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:44:17.335758 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.335727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw" event={"ID":"f2067629-41a1-4d38-8b75-5f232d8b8925","Type":"ContainerStarted","Data":"e13727276adc44886315db182c2981856af176d25721c5690f0ba950cdd25942"}
Sep 11 17:44:17.335867 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.335801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:17.371848 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.371789    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r" podStartSLOduration=2.482545507 podStartE2EDuration="6.371765546s" totalImagesPullingTime="3.889220039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:17.369024624 +0000 UTC m=+324.132479362" watchObservedRunningTime="2026-09-11 17:44:17.371765546 +0000 UTC m=+324.135220273"
Sep 11 17:44:17.396258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.396211    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw" podStartSLOduration=2.323451412 podStartE2EDuration="6.396190408s" totalImagesPullingTime="4.072738996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:17.393992707 +0000 UTC m=+324.157447434" watchObservedRunningTime="2026-09-11 17:44:17.396190408 +0000 UTC m=+324.159645138"
Sep 11 17:44:17.956517 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.956466    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:44:17.956624 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.956556    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:44:17.956624 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:17.956587    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:44:18.298614 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.298483    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"]
Sep 11 17:44:18.302510 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.302465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.305691 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.305664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 17:44:18.306026 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.306006    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rmfvs\""
Sep 11 17:44:18.306192 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.306177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 17:44:18.306345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.306329    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 17:44:18.315996 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.315969    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"]
Sep 11 17:44:18.342639 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.342606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh" event={"ID":"4b9f65a6-44d2-4bf1-8711-6939def63e7e","Type":"ContainerStarted","Data":"fe646468e430f74abedf9b29b83c8bf851cb0c1f325d48c146dfb2ad33598064"}
Sep 11 17:44:18.365411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.365362    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh" podStartSLOduration=2.036026608 podStartE2EDuration="6.36534623s" totalImagesPullingTime="4.329319622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:18.3638418 +0000 UTC m=+325.127296529" watchObservedRunningTime="2026-09-11 17:44:18.36534623 +0000 UTC m=+325.128801027"
Sep 11 17:44:18.423943 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.423903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/38c56a04-4af2-4457-962f-357212314f6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.424137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.423956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.424137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.424010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7p6\" (UniqueName: \"kubernetes.io/projected/38c56a04-4af2-4457-962f-357212314f6f-kube-api-access-vb7p6\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.424137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.424079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c56a04-4af2-4457-962f-357212314f6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.424304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.424223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.424723 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.424400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c56a04-4af2-4457-962f-357212314f6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.424723 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.424437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c56a04-4af2-4457-962f-357212314f6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/38c56a04-4af2-4457-962f-357212314f6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7p6\" (UniqueName: \"kubernetes.io/projected/38c56a04-4af2-4457-962f-357212314f6f-kube-api-access-vb7p6\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c56a04-4af2-4457-962f-357212314f6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.525531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.525511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.526177 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.526147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/38c56a04-4af2-4457-962f-357212314f6f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.528130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.528106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/38c56a04-4af2-4457-962f-357212314f6f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.528254 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.528170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.528581 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.528561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.528657 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.528592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38c56a04-4af2-4457-962f-357212314f6f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.535211 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.535164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7p6\" (UniqueName: \"kubernetes.io/projected/38c56a04-4af2-4457-962f-357212314f6f-kube-api-access-vb7p6\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.535211 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.535190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38c56a04-4af2-4457-962f-357212314f6f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-tdmbx\" (UID: \"38c56a04-4af2-4457-962f-357212314f6f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.615898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.615865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:18.901975 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:18.901951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"]
Sep 11 17:44:18.903616 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:44:18.903589    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c56a04_4af2_4457_962f_357212314f6f.slice/crio-a3758bdc2413fb61b468454f05ae696cd4902a7ac5562c607695fef2420c57a6 WatchSource:0}: Error finding container a3758bdc2413fb61b468454f05ae696cd4902a7ac5562c607695fef2420c57a6: Status 404 returned error can't find the container with id a3758bdc2413fb61b468454f05ae696cd4902a7ac5562c607695fef2420c57a6
Sep 11 17:44:19.303739 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:19.303659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:19.308380 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:19.308357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:19.358656 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:19.358582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx" event={"ID":"38c56a04-4af2-4457-962f-357212314f6f","Type":"ContainerStarted","Data":"a3758bdc2413fb61b468454f05ae696cd4902a7ac5562c607695fef2420c57a6"}
Sep 11 17:44:19.359119 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:19.358878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:19.359841 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:19.359816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh"
Sep 11 17:44:21.160104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:21.160070    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:44:21.160365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:21.160139    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:44:21.377167 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:21.377132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx" event={"ID":"38c56a04-4af2-4457-962f-357212314f6f","Type":"ContainerStarted","Data":"95b166a6005e5310223df6adf9391aad7a785265c9a63bbca0968720a763a6a5"}
Sep 11 17:44:21.377318 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:21.377230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:21.423227 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:21.423117    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx" podStartSLOduration=1.169261532 podStartE2EDuration="3.423097737s" totalImagesPullingTime="2.253836205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:21.42287041 +0000 UTC m=+328.186325137" watchObservedRunningTime="2026-09-11 17:44:21.423097737 +0000 UTC m=+328.186552463"
Sep 11 17:44:22.384163 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:22.384133    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-tdmbx"
Sep 11 17:44:28.345155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:28.345125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-qsqsw"
Sep 11 17:44:28.346827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:44:28.346809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vz62r"
Sep 11 17:45:12.348834 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:12.348749    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679c4568b5-jjdv7"]
Sep 11 17:45:16.451434 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.451398    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"]
Sep 11 17:45:16.454894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.454876    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.458103 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.458079    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:45:16.458208 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.458124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z7qc9\""
Sep 11 17:45:16.458273 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.458203    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:45:16.466326 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.466301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"]
Sep 11 17:45:16.547513 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.547455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.547698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.547545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.547698 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.547583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcq5\" (UniqueName: \"kubernetes.io/projected/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-kube-api-access-xfcq5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.548320 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.548301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"]
Sep 11 17:45:16.551767 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.551748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.562597 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.562576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"]
Sep 11 17:45:16.648559 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.648516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.648756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.648613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.648756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.648660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psr47\" (UniqueName: \"kubernetes.io/projected/5428d260-6bb1-46d8-b64b-afa663bce0e0-kube-api-access-psr47\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.648756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.648701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.648923 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.648822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.648923 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.648875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcq5\" (UniqueName: \"kubernetes.io/projected/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-kube-api-access-xfcq5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.649113 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.649088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.649295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.649133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.651250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.651231    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"]
Sep 11 17:45:16.654777 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.654763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.657636 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.657616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcq5\" (UniqueName: \"kubernetes.io/projected/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-kube-api-access-xfcq5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.666321 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.666296    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"]
Sep 11 17:45:16.749529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.749529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.749529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.749794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.749794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrff\" (UniqueName: \"kubernetes.io/projected/aa3c5b19-e5af-4557-be18-d50aa25cb290-kube-api-access-wsrff\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.749794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psr47\" (UniqueName: \"kubernetes.io/projected/5428d260-6bb1-46d8-b64b-afa663bce0e0-kube-api-access-psr47\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.749947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.749947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.749914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.750350 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.750332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"]
Sep 11 17:45:16.753903 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.753889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.763942 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.763915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:16.770671 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.770646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psr47\" (UniqueName: \"kubernetes.io/projected/5428d260-6bb1-46d8-b64b-afa663bce0e0-kube-api-access-psr47\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.836974 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.835042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"]
Sep 11 17:45:16.850434 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.850599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrff\" (UniqueName: \"kubernetes.io/projected/aa3c5b19-e5af-4557-be18-d50aa25cb290-kube-api-access-wsrff\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.850599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.850599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.850717 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hvg\" (UniqueName: \"kubernetes.io/projected/746a0835-8c7a-4dbc-9fa1-13ce89675133-kube-api-access-n4hvg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.850717 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.850891 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.851009 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.850991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.861370 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.861350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:16.869398 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.869377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrff\" (UniqueName: \"kubernetes.io/projected/aa3c5b19-e5af-4557-be18-d50aa25cb290-kube-api-access-wsrff\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:16.951583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.951550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.952395 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.952188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.952395 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.952220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hvg\" (UniqueName: \"kubernetes.io/projected/746a0835-8c7a-4dbc-9fa1-13ce89675133-kube-api-access-n4hvg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.952395 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.952351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.952697 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.952672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.962893 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.962861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hvg\" (UniqueName: \"kubernetes.io/projected/746a0835-8c7a-4dbc-9fa1-13ce89675133-kube-api-access-n4hvg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:16.972816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:16.972794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:17.039452 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.039426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"]
Sep 11 17:45:17.042161 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:45:17.042136    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2e2ca6_7ebf_4ba2_b9a4_a49a5b5fcd01.slice/crio-21bd124c690f56860c07bc6132bf7ffed5cddfbbc0ab6644ef7ce75ed55390a9 WatchSource:0}: Error finding container 21bd124c690f56860c07bc6132bf7ffed5cddfbbc0ab6644ef7ce75ed55390a9: Status 404 returned error can't find the container with id 21bd124c690f56860c07bc6132bf7ffed5cddfbbc0ab6644ef7ce75ed55390a9
Sep 11 17:45:17.062253 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.062229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:17.143752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.143700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"]
Sep 11 17:45:17.147745 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:45:17.147701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5428d260_6bb1_46d8_b64b_afa663bce0e0.slice/crio-e2342ce1e1aeac973dc56b975fac429d7fd3f6239246985dc0915641d792243f WatchSource:0}: Error finding container e2342ce1e1aeac973dc56b975fac429d7fd3f6239246985dc0915641d792243f: Status 404 returned error can't find the container with id e2342ce1e1aeac973dc56b975fac429d7fd3f6239246985dc0915641d792243f
Sep 11 17:45:17.248768 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.248742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"]
Sep 11 17:45:17.251806 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:45:17.251779    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3c5b19_e5af_4557_be18_d50aa25cb290.slice/crio-4db6552e56a030d3e33167436b250c47e64fa8bcbf9ea60dfcbd2d5c4ca108c1 WatchSource:0}: Error finding container 4db6552e56a030d3e33167436b250c47e64fa8bcbf9ea60dfcbd2d5c4ca108c1: Status 404 returned error can't find the container with id 4db6552e56a030d3e33167436b250c47e64fa8bcbf9ea60dfcbd2d5c4ca108c1
Sep 11 17:45:17.341033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.341004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"]
Sep 11 17:45:17.351052 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:45:17.351025    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746a0835_8c7a_4dbc_9fa1_13ce89675133.slice/crio-e7767abf7691e7ff686ef4d06d3a52a03d39386fc036a8de893bbaeee8d14f12 WatchSource:0}: Error finding container e7767abf7691e7ff686ef4d06d3a52a03d39386fc036a8de893bbaeee8d14f12: Status 404 returned error can't find the container with id e7767abf7691e7ff686ef4d06d3a52a03d39386fc036a8de893bbaeee8d14f12
Sep 11 17:45:17.647084 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.647046    2563 generic.go:356] "Generic (PLEG): container finished" podID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerID="d18b00b6b978a6ad6cd5fcbef50e804c6fb6a9b09a35e6eb04d3814a4bbee3d7" exitCode=0
Sep 11 17:45:17.647539 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.647116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm" event={"ID":"746a0835-8c7a-4dbc-9fa1-13ce89675133","Type":"ContainerDied","Data":"d18b00b6b978a6ad6cd5fcbef50e804c6fb6a9b09a35e6eb04d3814a4bbee3d7"}
Sep 11 17:45:17.647539 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.647178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm" event={"ID":"746a0835-8c7a-4dbc-9fa1-13ce89675133","Type":"ContainerStarted","Data":"e7767abf7691e7ff686ef4d06d3a52a03d39386fc036a8de893bbaeee8d14f12"}
Sep 11 17:45:17.648729 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.648707    2563 generic.go:356] "Generic (PLEG): container finished" podID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerID="73ec3ffabf28c28261180296bdad6ff8f66763d8e2eab5f3a324e815ea8f2eb2" exitCode=0
Sep 11 17:45:17.648814 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.648795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6" event={"ID":"5428d260-6bb1-46d8-b64b-afa663bce0e0","Type":"ContainerDied","Data":"73ec3ffabf28c28261180296bdad6ff8f66763d8e2eab5f3a324e815ea8f2eb2"}
Sep 11 17:45:17.648914 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.648831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6" event={"ID":"5428d260-6bb1-46d8-b64b-afa663bce0e0","Type":"ContainerStarted","Data":"e2342ce1e1aeac973dc56b975fac429d7fd3f6239246985dc0915641d792243f"}
Sep 11 17:45:17.650390 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.650371    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerID="8bc011d7e2d728e81a4173354bcf0f6aa2792c5cb83394cb177dcf30f25120ee" exitCode=0
Sep 11 17:45:17.650468 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.650444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg" event={"ID":"aa3c5b19-e5af-4557-be18-d50aa25cb290","Type":"ContainerDied","Data":"8bc011d7e2d728e81a4173354bcf0f6aa2792c5cb83394cb177dcf30f25120ee"}
Sep 11 17:45:17.650550 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.650480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg" event={"ID":"aa3c5b19-e5af-4557-be18-d50aa25cb290","Type":"ContainerStarted","Data":"4db6552e56a030d3e33167436b250c47e64fa8bcbf9ea60dfcbd2d5c4ca108c1"}
Sep 11 17:45:17.652050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.652030    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerID="aac0b006fab8799ff801650e4b072f22398c87b6d5284370a51096da978108e4" exitCode=0
Sep 11 17:45:17.652120 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.652073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n" event={"ID":"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01","Type":"ContainerDied","Data":"aac0b006fab8799ff801650e4b072f22398c87b6d5284370a51096da978108e4"}
Sep 11 17:45:17.652120 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:17.652101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n" event={"ID":"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01","Type":"ContainerStarted","Data":"21bd124c690f56860c07bc6132bf7ffed5cddfbbc0ab6644ef7ce75ed55390a9"}
Sep 11 17:45:19.665989 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.665963    2563 generic.go:356] "Generic (PLEG): container finished" podID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerID="07ce1c3721abe389ab750b015e8e4c0a19065bdc19f9e4bd6e3be551be3afb85" exitCode=0
Sep 11 17:45:19.666369 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.666034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm" event={"ID":"746a0835-8c7a-4dbc-9fa1-13ce89675133","Type":"ContainerDied","Data":"07ce1c3721abe389ab750b015e8e4c0a19065bdc19f9e4bd6e3be551be3afb85"}
Sep 11 17:45:19.668277 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.668249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6" event={"ID":"5428d260-6bb1-46d8-b64b-afa663bce0e0","Type":"ContainerStarted","Data":"cfb4a166fffb40ddaab540c3acdaaff608e192d68a0d53dde8c1b1e0602a9a92"}
Sep 11 17:45:19.670235 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.670209    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerID="8abdb357c040f0d6640d880cc382d2e14593bb77b221d8f83301abab2efe457c" exitCode=0
Sep 11 17:45:19.670345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.670283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg" event={"ID":"aa3c5b19-e5af-4557-be18-d50aa25cb290","Type":"ContainerDied","Data":"8abdb357c040f0d6640d880cc382d2e14593bb77b221d8f83301abab2efe457c"}
Sep 11 17:45:19.672805 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.672783    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerID="f63c212e6217588321efbae40693df994f91f4899ded5eff171b7bfed124649b" exitCode=0
Sep 11 17:45:19.672805 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:19.672816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n" event={"ID":"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01","Type":"ContainerDied","Data":"f63c212e6217588321efbae40693df994f91f4899ded5eff171b7bfed124649b"}
Sep 11 17:45:20.680322 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.680285    2563 generic.go:356] "Generic (PLEG): container finished" podID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerID="3339e036e48f3b6a6a801b20dc8489c0f43d62b0de8c1701064118f3ed4fc30c" exitCode=0
Sep 11 17:45:20.680777 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.680367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm" event={"ID":"746a0835-8c7a-4dbc-9fa1-13ce89675133","Type":"ContainerDied","Data":"3339e036e48f3b6a6a801b20dc8489c0f43d62b0de8c1701064118f3ed4fc30c"}
Sep 11 17:45:20.682133 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.682106    2563 generic.go:356] "Generic (PLEG): container finished" podID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerID="cfb4a166fffb40ddaab540c3acdaaff608e192d68a0d53dde8c1b1e0602a9a92" exitCode=0
Sep 11 17:45:20.682245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.682183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6" event={"ID":"5428d260-6bb1-46d8-b64b-afa663bce0e0","Type":"ContainerDied","Data":"cfb4a166fffb40ddaab540c3acdaaff608e192d68a0d53dde8c1b1e0602a9a92"}
Sep 11 17:45:20.685666 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.684884    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerID="0a8261f577f0c2687789a434cc56704b0d0f03b4303cf57906a1b64b32d94d30" exitCode=0
Sep 11 17:45:20.685666 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.685034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg" event={"ID":"aa3c5b19-e5af-4557-be18-d50aa25cb290","Type":"ContainerDied","Data":"0a8261f577f0c2687789a434cc56704b0d0f03b4303cf57906a1b64b32d94d30"}
Sep 11 17:45:20.692557 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.692529    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerID="4be9ddbbefb49994d162143dbba661d959c2ac3128a89dffc6067d6235520e82" exitCode=0
Sep 11 17:45:20.692641 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:20.692570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n" event={"ID":"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01","Type":"ContainerDied","Data":"4be9ddbbefb49994d162143dbba661d959c2ac3128a89dffc6067d6235520e82"}
Sep 11 17:45:21.699945 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.699902    2563 generic.go:356] "Generic (PLEG): container finished" podID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerID="7b12e1481f4e171ed6f5a9d742acc525418b5ebfde8348ea6662b195abbb9bee" exitCode=0
Sep 11 17:45:21.700331 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.699962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6" event={"ID":"5428d260-6bb1-46d8-b64b-afa663bce0e0","Type":"ContainerDied","Data":"7b12e1481f4e171ed6f5a9d742acc525418b5ebfde8348ea6662b195abbb9bee"}
Sep 11 17:45:21.849374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.849346    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:21.882829 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.882805    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:21.885816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.885792    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:21.999816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-util\") pod \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") "
Sep 11 17:45:21.999816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-util\") pod \"746a0835-8c7a-4dbc-9fa1-13ce89675133\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") "
Sep 11 17:45:21.999816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-bundle\") pod \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") "
Sep 11 17:45:22.000102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-util\") pod \"aa3c5b19-e5af-4557-be18-d50aa25cb290\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") "
Sep 11 17:45:22.000102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfcq5\" (UniqueName: \"kubernetes.io/projected/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-kube-api-access-xfcq5\") pod \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\" (UID: \"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01\") "
Sep 11 17:45:22.000102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-bundle\") pod \"746a0835-8c7a-4dbc-9fa1-13ce89675133\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") "
Sep 11 17:45:22.000102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:21.999976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsrff\" (UniqueName: \"kubernetes.io/projected/aa3c5b19-e5af-4557-be18-d50aa25cb290-kube-api-access-wsrff\") pod \"aa3c5b19-e5af-4557-be18-d50aa25cb290\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") "
Sep 11 17:45:22.000102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.000012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-bundle\") pod \"aa3c5b19-e5af-4557-be18-d50aa25cb290\" (UID: \"aa3c5b19-e5af-4557-be18-d50aa25cb290\") "
Sep 11 17:45:22.000102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.000046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4hvg\" (UniqueName: \"kubernetes.io/projected/746a0835-8c7a-4dbc-9fa1-13ce89675133-kube-api-access-n4hvg\") pod \"746a0835-8c7a-4dbc-9fa1-13ce89675133\" (UID: \"746a0835-8c7a-4dbc-9fa1-13ce89675133\") "
Sep 11 17:45:22.000717 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.000512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-bundle" (OuterVolumeSpecName: "bundle") pod "6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" (UID: "6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.001084 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.001045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-bundle" (OuterVolumeSpecName: "bundle") pod "746a0835-8c7a-4dbc-9fa1-13ce89675133" (UID: "746a0835-8c7a-4dbc-9fa1-13ce89675133"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.001220 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.001148    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-bundle" (OuterVolumeSpecName: "bundle") pod "aa3c5b19-e5af-4557-be18-d50aa25cb290" (UID: "aa3c5b19-e5af-4557-be18-d50aa25cb290"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.002779 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.002751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-kube-api-access-xfcq5" (OuterVolumeSpecName: "kube-api-access-xfcq5") pod "6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" (UID: "6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01"). InnerVolumeSpecName "kube-api-access-xfcq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:22.002880 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.002796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746a0835-8c7a-4dbc-9fa1-13ce89675133-kube-api-access-n4hvg" (OuterVolumeSpecName: "kube-api-access-n4hvg") pod "746a0835-8c7a-4dbc-9fa1-13ce89675133" (UID: "746a0835-8c7a-4dbc-9fa1-13ce89675133"). InnerVolumeSpecName "kube-api-access-n4hvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:22.003004 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.002967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3c5b19-e5af-4557-be18-d50aa25cb290-kube-api-access-wsrff" (OuterVolumeSpecName: "kube-api-access-wsrff") pod "aa3c5b19-e5af-4557-be18-d50aa25cb290" (UID: "aa3c5b19-e5af-4557-be18-d50aa25cb290"). InnerVolumeSpecName "kube-api-access-wsrff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:22.006653 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.006617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-util" (OuterVolumeSpecName: "util") pod "746a0835-8c7a-4dbc-9fa1-13ce89675133" (UID: "746a0835-8c7a-4dbc-9fa1-13ce89675133"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.006802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.006785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-util" (OuterVolumeSpecName: "util") pod "6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" (UID: "6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.008599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.008576    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-util" (OuterVolumeSpecName: "util") pod "aa3c5b19-e5af-4557-be18-d50aa25cb290" (UID: "aa3c5b19-e5af-4557-be18-d50aa25cb290"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.101830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101786    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsrff\" (UniqueName: \"kubernetes.io/projected/aa3c5b19-e5af-4557-be18-d50aa25cb290-kube-api-access-wsrff\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.101830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101829    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.101830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101842    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hvg\" (UniqueName: \"kubernetes.io/projected/746a0835-8c7a-4dbc-9fa1-13ce89675133-kube-api-access-n4hvg\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.102071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101851    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.102071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101860    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.102071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101867    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.102071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101874    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa3c5b19-e5af-4557-be18-d50aa25cb290-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.102071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101882    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcq5\" (UniqueName: \"kubernetes.io/projected/6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01-kube-api-access-xfcq5\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.102071 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.101890    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/746a0835-8c7a-4dbc-9fa1-13ce89675133-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:22.707050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.707016    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm"
Sep 11 17:45:22.707436 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.707025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajrdlm" event={"ID":"746a0835-8c7a-4dbc-9fa1-13ce89675133","Type":"ContainerDied","Data":"e7767abf7691e7ff686ef4d06d3a52a03d39386fc036a8de893bbaeee8d14f12"}
Sep 11 17:45:22.707436 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.707139    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7767abf7691e7ff686ef4d06d3a52a03d39386fc036a8de893bbaeee8d14f12"
Sep 11 17:45:22.709093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.709079    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg"
Sep 11 17:45:22.709169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.709079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7s8bkg" event={"ID":"aa3c5b19-e5af-4557-be18-d50aa25cb290","Type":"ContainerDied","Data":"4db6552e56a030d3e33167436b250c47e64fa8bcbf9ea60dfcbd2d5c4ca108c1"}
Sep 11 17:45:22.709224 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.709191    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db6552e56a030d3e33167436b250c47e64fa8bcbf9ea60dfcbd2d5c4ca108c1"
Sep 11 17:45:22.711293 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.711274    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n"
Sep 11 17:45:22.711386 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.711295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdrl4n" event={"ID":"6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01","Type":"ContainerDied","Data":"21bd124c690f56860c07bc6132bf7ffed5cddfbbc0ab6644ef7ce75ed55390a9"}
Sep 11 17:45:22.711386 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.711333    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21bd124c690f56860c07bc6132bf7ffed5cddfbbc0ab6644ef7ce75ed55390a9"
Sep 11 17:45:22.828251 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.828226    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:22.909549 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.909483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-util\") pod \"5428d260-6bb1-46d8-b64b-afa663bce0e0\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") "
Sep 11 17:45:22.909715 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.909590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psr47\" (UniqueName: \"kubernetes.io/projected/5428d260-6bb1-46d8-b64b-afa663bce0e0-kube-api-access-psr47\") pod \"5428d260-6bb1-46d8-b64b-afa663bce0e0\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") "
Sep 11 17:45:22.909715 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.909645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-bundle\") pod \"5428d260-6bb1-46d8-b64b-afa663bce0e0\" (UID: \"5428d260-6bb1-46d8-b64b-afa663bce0e0\") "
Sep 11 17:45:22.910084 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.910063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-bundle" (OuterVolumeSpecName: "bundle") pod "5428d260-6bb1-46d8-b64b-afa663bce0e0" (UID: "5428d260-6bb1-46d8-b64b-afa663bce0e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:22.911920 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.911894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5428d260-6bb1-46d8-b64b-afa663bce0e0-kube-api-access-psr47" (OuterVolumeSpecName: "kube-api-access-psr47") pod "5428d260-6bb1-46d8-b64b-afa663bce0e0" (UID: "5428d260-6bb1-46d8-b64b-afa663bce0e0"). InnerVolumeSpecName "kube-api-access-psr47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:22.914209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:22.914173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-util" (OuterVolumeSpecName: "util") pod "5428d260-6bb1-46d8-b64b-afa663bce0e0" (UID: "5428d260-6bb1-46d8-b64b-afa663bce0e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:23.011283 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:23.011202    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psr47\" (UniqueName: \"kubernetes.io/projected/5428d260-6bb1-46d8-b64b-afa663bce0e0-kube-api-access-psr47\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:23.011283 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:23.011229    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:23.011283 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:23.011238    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5428d260-6bb1-46d8-b64b-afa663bce0e0-util\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:23.718197 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:23.718171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6"
Sep 11 17:45:23.718197 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:23.718195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nblg6" event={"ID":"5428d260-6bb1-46d8-b64b-afa663bce0e0","Type":"ContainerDied","Data":"e2342ce1e1aeac973dc56b975fac429d7fd3f6239246985dc0915641d792243f"}
Sep 11 17:45:23.718729 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:23.718233    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2342ce1e1aeac973dc56b975fac429d7fd3f6239246985dc0915641d792243f"
Sep 11 17:45:37.368390 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.368323    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679c4568b5-jjdv7" podUID="ed7597b1-62b0-4131-bee6-d9564147bc9f" containerName="console" containerID="cri-o://ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7" gracePeriod=15
Sep 11 17:45:37.620112 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.620049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:45:37.737506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhp9d\" (UniqueName: \"kubernetes.io/projected/ed7597b1-62b0-4131-bee6-d9564147bc9f-kube-api-access-lhp9d\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.737660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-trusted-ca-bundle\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.737660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-oauth-serving-cert\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.737660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-service-ca\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.737660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737624    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-config\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.737660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-serving-cert\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.737928 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.737852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-oauth-config\") pod \"ed7597b1-62b0-4131-bee6-d9564147bc9f\" (UID: \"ed7597b1-62b0-4131-bee6-d9564147bc9f\") "
Sep 11 17:45:37.738077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-config" (OuterVolumeSpecName: "console-config") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:37.738077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:37.738077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:37.738077 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-service-ca" (OuterVolumeSpecName: "service-ca") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:45:37.738311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738161    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-trusted-ca-bundle\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:37.738311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738173    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-oauth-serving-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:37.738311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738182    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:37.738311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.738192    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:37.739941 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.739915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:37.740040 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.739943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7597b1-62b0-4131-bee6-d9564147bc9f-kube-api-access-lhp9d" (OuterVolumeSpecName: "kube-api-access-lhp9d") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "kube-api-access-lhp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:37.740258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.740240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ed7597b1-62b0-4131-bee6-d9564147bc9f" (UID: "ed7597b1-62b0-4131-bee6-d9564147bc9f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:37.792187 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.792158    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed7597b1-62b0-4131-bee6-d9564147bc9f" containerID="ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7" exitCode=0
Sep 11 17:45:37.792311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.792198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679c4568b5-jjdv7"
Sep 11 17:45:37.792311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.792237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679c4568b5-jjdv7" event={"ID":"ed7597b1-62b0-4131-bee6-d9564147bc9f","Type":"ContainerDied","Data":"ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7"}
Sep 11 17:45:37.792311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.792286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679c4568b5-jjdv7" event={"ID":"ed7597b1-62b0-4131-bee6-d9564147bc9f","Type":"ContainerDied","Data":"3571c2daa3304b80025181682e9b1710a9b80ef8aed6be5426c956e3a5b7d1de"}
Sep 11 17:45:37.792311 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.792306    2563 scope.go:117] "RemoveContainer" containerID="ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7"
Sep 11 17:45:37.801045 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.801030    2563 scope.go:117] "RemoveContainer" containerID="ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7"
Sep 11 17:45:37.801323 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:45:37.801304    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7\": container with ID starting with ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7 not found: ID does not exist" containerID="ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7"
Sep 11 17:45:37.801364 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.801332    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7"} err="failed to get container status \"ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7\": rpc error: code = NotFound desc = could not find container \"ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7\": container with ID starting with ad62e72f708fac0f6e523b02584ffa692b9e530d366f1d7e08e26bd449112fa7 not found: ID does not exist"
Sep 11 17:45:37.817387 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.817364    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679c4568b5-jjdv7"]
Sep 11 17:45:37.821858 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.821840    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679c4568b5-jjdv7"]
Sep 11 17:45:37.839042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.839017    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-serving-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:37.839042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.839038    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed7597b1-62b0-4131-bee6-d9564147bc9f-console-oauth-config\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:37.839163 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:37.839048    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp9d\" (UniqueName: \"kubernetes.io/projected/ed7597b1-62b0-4131-bee6-d9564147bc9f-kube-api-access-lhp9d\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:45:39.780545 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:45:39.780514    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7597b1-62b0-4131-bee6-d9564147bc9f" path="/var/lib/kubelet/pods/ed7597b1-62b0-4131-bee6-d9564147bc9f/volumes"
Sep 11 17:46:27.595340 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595248    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"]
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595783    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="util"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="util"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595813    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="pull"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595821    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="pull"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595833    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="extract"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595842    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="extract"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="util"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595866    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="util"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595877    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="extract"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595885    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="extract"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="pull"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595906    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="pull"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595917    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7597b1-62b0-4131-bee6-d9564147bc9f" containerName="console"
Sep 11 17:46:27.595924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595924    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7597b1-62b0-4131-bee6-d9564147bc9f" containerName="console"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595934    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="util"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595952    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="util"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595959    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="pull"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="pull"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595978    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="pull"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595986    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="pull"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.595994    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596002    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="util"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596021    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="util"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596030    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596038    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596133    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="746a0835-8c7a-4dbc-9fa1-13ce89675133" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596145    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2e2ca6-7ebf-4ba2-b9a4-a49a5b5fcd01" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596157    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3c5b19-e5af-4557-be18-d50aa25cb290" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596165    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5428d260-6bb1-46d8-b64b-afa663bce0e0" containerName="extract"
Sep 11 17:46:27.596625 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.596173    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7597b1-62b0-4131-bee6-d9564147bc9f" containerName="console"
Sep 11 17:46:27.599304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.599283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.602198 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.602180    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cp8h7\""
Sep 11 17:46:27.610792 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.610765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"]
Sep 11 17:46:27.676904 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.676860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769ab66a-cade-43e5-a337-c6697aaa6aca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.676904 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.676907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.676940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.676978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.677044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.677085    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.677117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.677142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.677345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.677165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj466\" (UniqueName: \"kubernetes.io/projected/769ab66a-cade-43e5-a337-c6697aaa6aca-kube-api-access-kj466\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778384 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778610 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778610 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj466\" (UniqueName: \"kubernetes.io/projected/769ab66a-cade-43e5-a337-c6697aaa6aca-kube-api-access-kj466\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778610 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769ab66a-cade-43e5-a337-c6697aaa6aca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778610 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778610 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778610 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778916 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.778977 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.778920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.779052 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.779018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.779114 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.779077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.779292 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.779264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769ab66a-cade-43e5-a337-c6697aaa6aca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.779393 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.779277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.779393 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.779337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.784015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.781249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.786341 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.786315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.788373 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.788337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769ab66a-cade-43e5-a337-c6697aaa6aca-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.789638 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.789616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj466\" (UniqueName: \"kubernetes.io/projected/769ab66a-cade-43e5-a337-c6697aaa6aca-kube-api-access-kj466\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4grfw\" (UID: \"769ab66a-cade-43e5-a337-c6697aaa6aca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:27.910452 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:27.910359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:28.188164 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:28.188138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"]
Sep 11 17:46:28.190525 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:46:28.190474    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769ab66a_cade_43e5_a337_c6697aaa6aca.slice/crio-db6f1f58e16e67ff2ca65fea36fa7dedb1f9176dd7bf28788f3daccbba345ff3 WatchSource:0}: Error finding container db6f1f58e16e67ff2ca65fea36fa7dedb1f9176dd7bf28788f3daccbba345ff3: Status 404 returned error can't find the container with id db6f1f58e16e67ff2ca65fea36fa7dedb1f9176dd7bf28788f3daccbba345ff3
Sep 11 17:46:28.193035 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:28.193002    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:46:28.193122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:28.193071    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:46:28.193122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:28.193098    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:46:29.055010 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:29.054972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" event={"ID":"769ab66a-cade-43e5-a337-c6697aaa6aca","Type":"ContainerStarted","Data":"3fbfcda9bddbb38f0b2ccb090793fc2cb59a6426cc319021d549583aa7210df8"}
Sep 11 17:46:29.055409 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:29.055029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" event={"ID":"769ab66a-cade-43e5-a337-c6697aaa6aca","Type":"ContainerStarted","Data":"db6f1f58e16e67ff2ca65fea36fa7dedb1f9176dd7bf28788f3daccbba345ff3"}
Sep 11 17:46:29.080842 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:29.080791    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" podStartSLOduration=2.080775794 podStartE2EDuration="2.080775794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:29.07888558 +0000 UTC m=+455.842340311" watchObservedRunningTime="2026-09-11 17:46:29.080775794 +0000 UTC m=+455.844230521"
Sep 11 17:46:29.911408 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:29.911374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:29.916430 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:29.916405    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:30.059770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:30.059743    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:46:30.060827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:46:30.060806    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:47:15.669375 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.669345    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5949d9874c-7g9bz"]
Sep 11 17:47:15.672544 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.672528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.677628 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.677607    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 17:47:15.678471 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.678423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tjzmj\""
Sep 11 17:47:15.678620 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.678526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 17:47:15.678620 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.678531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 17:47:15.702165 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.702123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-webhook-cert\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.702347 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.702187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fll5q\" (UniqueName: \"kubernetes.io/projected/a77f3d83-5732-4965-9053-d2a4b04c8435-kube-api-access-fll5q\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.702347 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.702285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.703593 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.703556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5949d9874c-7g9bz"]
Sep 11 17:47:15.803561 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.803526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-webhook-cert\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.803809 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.803688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fll5q\" (UniqueName: \"kubernetes.io/projected/a77f3d83-5732-4965-9053-d2a4b04c8435-kube-api-access-fll5q\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.803888 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.803870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.804010 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:47:15.803990    2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 11 17:47:15.804104 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:47:15.804060    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs podName:a77f3d83-5732-4965-9053-d2a4b04c8435 nodeName:}" failed. No retries permitted until 2026-09-11 17:47:16.304040528 +0000 UTC m=+503.067495235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs") pod "maas-controller-5949d9874c-7g9bz" (UID: "a77f3d83-5732-4965-9053-d2a4b04c8435") : secret "maas-controller-metrics-tls" not found
Sep 11 17:47:15.806152 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.806134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-webhook-cert\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.815199 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.815167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fll5q\" (UniqueName: \"kubernetes.io/projected/a77f3d83-5732-4965-9053-d2a4b04c8435-kube-api-access-fll5q\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:15.950047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.949965    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b57799854-rbrp8"]
Sep 11 17:47:15.953271 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.953253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:15.989500 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:15.989453    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b57799854-rbrp8"]
Sep 11 17:47:16.006069 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.006039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-webhook-cert\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.006250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.006082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-metrics-certs\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.006250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.006207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mxf\" (UniqueName: \"kubernetes.io/projected/39e1ea4e-773d-4637-aaba-bb74230a4d45-kube-api-access-z6mxf\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.107551 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.107479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-webhook-cert\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.107551 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.107548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-metrics-certs\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.107750 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.107597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mxf\" (UniqueName: \"kubernetes.io/projected/39e1ea4e-773d-4637-aaba-bb74230a4d45-kube-api-access-z6mxf\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.110159 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.110134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-webhook-cert\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.110290 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.110266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-metrics-certs\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.136753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.136724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mxf\" (UniqueName: \"kubernetes.io/projected/39e1ea4e-773d-4637-aaba-bb74230a4d45-kube-api-access-z6mxf\") pod \"maas-controller-5b57799854-rbrp8\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") " pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.263435 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.263350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:16.309910 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.309870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:16.312459 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.312434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs\") pod \"maas-controller-5949d9874c-7g9bz\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") " pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:16.551644 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.551551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b57799854-rbrp8"]
Sep 11 17:47:16.556987 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:47:16.556960    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e1ea4e_773d_4637_aaba_bb74230a4d45.slice/crio-78bf1a7f9ece257746aaa2c38cf630afe961b5b01bb837912383664bd1c58434 WatchSource:0}: Error finding container 78bf1a7f9ece257746aaa2c38cf630afe961b5b01bb837912383664bd1c58434: Status 404 returned error can't find the container with id 78bf1a7f9ece257746aaa2c38cf630afe961b5b01bb837912383664bd1c58434
Sep 11 17:47:16.583263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.583232    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:16.866696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:16.866580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5949d9874c-7g9bz"]
Sep 11 17:47:16.869509 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:47:16.869458    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77f3d83_5732_4965_9053_d2a4b04c8435.slice/crio-df9b64b79d46b54fea6e17172284211b62843bb591c63ae93f111b9ec489e0dd WatchSource:0}: Error finding container df9b64b79d46b54fea6e17172284211b62843bb591c63ae93f111b9ec489e0dd: Status 404 returned error can't find the container with id df9b64b79d46b54fea6e17172284211b62843bb591c63ae93f111b9ec489e0dd
Sep 11 17:47:17.302343 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:17.302207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5949d9874c-7g9bz" event={"ID":"a77f3d83-5732-4965-9053-d2a4b04c8435","Type":"ContainerStarted","Data":"df9b64b79d46b54fea6e17172284211b62843bb591c63ae93f111b9ec489e0dd"}
Sep 11 17:47:17.304413 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:17.304382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-rbrp8" event={"ID":"39e1ea4e-773d-4637-aaba-bb74230a4d45","Type":"ContainerStarted","Data":"78bf1a7f9ece257746aaa2c38cf630afe961b5b01bb837912383664bd1c58434"}
Sep 11 17:47:19.320064 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:19.320030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5949d9874c-7g9bz" event={"ID":"a77f3d83-5732-4965-9053-d2a4b04c8435","Type":"ContainerStarted","Data":"6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba"}
Sep 11 17:47:19.320527 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:19.320091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:19.321623 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:19.321596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-rbrp8" event={"ID":"39e1ea4e-773d-4637-aaba-bb74230a4d45","Type":"ContainerStarted","Data":"a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9"}
Sep 11 17:47:19.321748 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:19.321735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:19.378848 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:19.378794    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5949d9874c-7g9bz" podStartSLOduration=2.050737998 podStartE2EDuration="4.378777439s" totalImagesPullingTime="2.328039441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:19.377262906 +0000 UTC m=+506.140717633" watchObservedRunningTime="2026-09-11 17:47:19.378777439 +0000 UTC m=+506.142232166"
Sep 11 17:47:19.437323 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:19.437216    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b57799854-rbrp8" podStartSLOduration=1.807319835 podStartE2EDuration="4.437198073s" totalImagesPullingTime="2.629878238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:19.436178082 +0000 UTC m=+506.199632809" watchObservedRunningTime="2026-09-11 17:47:19.437198073 +0000 UTC m=+506.200652800"
Sep 11 17:47:30.329248 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.329215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:30.329747 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.329385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 17:47:30.398335 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.398296    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5949d9874c-7g9bz"]
Sep 11 17:47:30.398637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.398609    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5949d9874c-7g9bz" podUID="a77f3d83-5732-4965-9053-d2a4b04c8435" containerName="manager" containerID="cri-o://6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba" gracePeriod=30
Sep 11 17:47:30.633258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.633230    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:30.741554 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.741525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs\") pod \"a77f3d83-5732-4965-9053-d2a4b04c8435\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") "
Sep 11 17:47:30.741728 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.741581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-webhook-cert\") pod \"a77f3d83-5732-4965-9053-d2a4b04c8435\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") "
Sep 11 17:47:30.741728 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.741603    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fll5q\" (UniqueName: \"kubernetes.io/projected/a77f3d83-5732-4965-9053-d2a4b04c8435-kube-api-access-fll5q\") pod \"a77f3d83-5732-4965-9053-d2a4b04c8435\" (UID: \"a77f3d83-5732-4965-9053-d2a4b04c8435\") "
Sep 11 17:47:30.743972 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.743943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a77f3d83-5732-4965-9053-d2a4b04c8435" (UID: "a77f3d83-5732-4965-9053-d2a4b04c8435"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:47:30.743972 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.743962    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a77f3d83-5732-4965-9053-d2a4b04c8435" (UID: "a77f3d83-5732-4965-9053-d2a4b04c8435"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:47:30.744137 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.744016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77f3d83-5732-4965-9053-d2a4b04c8435-kube-api-access-fll5q" (OuterVolumeSpecName: "kube-api-access-fll5q") pod "a77f3d83-5732-4965-9053-d2a4b04c8435" (UID: "a77f3d83-5732-4965-9053-d2a4b04c8435"). InnerVolumeSpecName "kube-api-access-fll5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:47:30.842837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.842794    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:47:30.842837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.842832    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a77f3d83-5732-4965-9053-d2a4b04c8435-webhook-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:47:30.842837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:30.842846    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fll5q\" (UniqueName: \"kubernetes.io/projected/a77f3d83-5732-4965-9053-d2a4b04c8435-kube-api-access-fll5q\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:47:31.386660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.386625    2563 generic.go:356] "Generic (PLEG): container finished" podID="a77f3d83-5732-4965-9053-d2a4b04c8435" containerID="6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba" exitCode=0
Sep 11 17:47:31.387095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.386675    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5949d9874c-7g9bz"
Sep 11 17:47:31.387095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.386706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5949d9874c-7g9bz" event={"ID":"a77f3d83-5732-4965-9053-d2a4b04c8435","Type":"ContainerDied","Data":"6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba"}
Sep 11 17:47:31.387095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.386767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5949d9874c-7g9bz" event={"ID":"a77f3d83-5732-4965-9053-d2a4b04c8435","Type":"ContainerDied","Data":"df9b64b79d46b54fea6e17172284211b62843bb591c63ae93f111b9ec489e0dd"}
Sep 11 17:47:31.387095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.386796    2563 scope.go:117] "RemoveContainer" containerID="6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba"
Sep 11 17:47:31.396897 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.396880    2563 scope.go:117] "RemoveContainer" containerID="6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba"
Sep 11 17:47:31.397188 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:47:31.397170    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba\": container with ID starting with 6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba not found: ID does not exist" containerID="6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba"
Sep 11 17:47:31.397242 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.397200    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba"} err="failed to get container status \"6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba\": rpc error: code = NotFound desc = could not find container \"6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba\": container with ID starting with 6f6d51ad64cefeacd021ad78e98ab6519b46681abf0b81ee9742f0e909ba7eba not found: ID does not exist"
Sep 11 17:47:31.411556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.411530    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5949d9874c-7g9bz"]
Sep 11 17:47:31.416807 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.416780    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5949d9874c-7g9bz"]
Sep 11 17:47:31.781476 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:31.781382    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77f3d83-5732-4965-9053-d2a4b04c8435" path="/var/lib/kubelet/pods/a77f3d83-5732-4965-9053-d2a4b04c8435/volumes"
Sep 11 17:47:53.528525 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.528414    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"]
Sep 11 17:47:53.528913 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.528847    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77f3d83-5732-4965-9053-d2a4b04c8435" containerName="manager"
Sep 11 17:47:53.528913 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.528859    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77f3d83-5732-4965-9053-d2a4b04c8435" containerName="manager"
Sep 11 17:47:53.528981 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.528920    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77f3d83-5732-4965-9053-d2a4b04c8435" containerName="manager"
Sep 11 17:47:53.533359 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.533341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.536764 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.536733    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 17:47:53.537042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.537026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5vrxn\""
Sep 11 17:47:53.537116 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.537061    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 17:47:53.538160 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.538142    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:47:53.538274 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.538168    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 17:47:53.548547 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.548521    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"]
Sep 11 17:47:53.643687 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.643656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frgb8\" (UniqueName: \"kubernetes.io/projected/1923587f-4658-4c84-809f-e9984da40fbb-kube-api-access-frgb8\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.643687 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.643689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-metrics-certs\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.643897 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.643761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.643897 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.643776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923587f-4658-4c84-809f-e9984da40fbb-openshift-service-ca\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.744846 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.744817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frgb8\" (UniqueName: \"kubernetes.io/projected/1923587f-4658-4c84-809f-e9984da40fbb-kube-api-access-frgb8\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.744973 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.744852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-metrics-certs\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.745040 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.744981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.745040 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.745001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923587f-4658-4c84-809f-e9984da40fbb-openshift-service-ca\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.748257 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.748237    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 17:47:53.748397 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.748380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:47:53.748476 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.748460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 17:47:53.755276 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:47:53.755248    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 11 17:47:53.755418 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:47:53.755353    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls podName:1923587f-4658-4c84-809f-e9984da40fbb nodeName:}" failed. No retries permitted until 2026-09-11 17:47:54.255332034 +0000 UTC m=+541.018786758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls") pod "maas-api-579ffb46f9-4pv54" (UID: "1923587f-4658-4c84-809f-e9984da40fbb") : secret "maas-api-serving-cert" not found
Sep 11 17:47:53.756187 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.756165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923587f-4658-4c84-809f-e9984da40fbb-openshift-service-ca\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.758258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.758236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-metrics-certs\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:53.760720 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.760699    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 17:47:53.771129 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:53.771105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frgb8\" (UniqueName: \"kubernetes.io/projected/1923587f-4658-4c84-809f-e9984da40fbb-kube-api-access-frgb8\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:54.350551 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:54.350512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:54.353069 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:54.353048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls\") pod \"maas-api-579ffb46f9-4pv54\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") " pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:54.447329 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:54.447298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5vrxn\""
Sep 11 17:47:54.454432 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:54.454405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:54.728994 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:54.728971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"]
Sep 11 17:47:54.731388 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:47:54.731357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1923587f_4658_4c84_809f_e9984da40fbb.slice/crio-79ad31ecf543e90003083796a4416281792c45710972989b590f8d8be0f090d6 WatchSource:0}: Error finding container 79ad31ecf543e90003083796a4416281792c45710972989b590f8d8be0f090d6: Status 404 returned error can't find the container with id 79ad31ecf543e90003083796a4416281792c45710972989b590f8d8be0f090d6
Sep 11 17:47:55.517667 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:55.517631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54" event={"ID":"1923587f-4658-4c84-809f-e9984da40fbb","Type":"ContainerStarted","Data":"79ad31ecf543e90003083796a4416281792c45710972989b590f8d8be0f090d6"}
Sep 11 17:47:57.536224 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:57.536183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54" event={"ID":"1923587f-4658-4c84-809f-e9984da40fbb","Type":"ContainerStarted","Data":"588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484"}
Sep 11 17:47:57.536685 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:57.536342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:47:57.576333 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:47:57.576283    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54" podStartSLOduration=2.280633405 podStartE2EDuration="4.576264881s" totalImagesPullingTime="2.295631476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:57.575287219 +0000 UTC m=+544.338741948" watchObservedRunningTime="2026-09-11 17:47:57.576264881 +0000 UTC m=+544.339719609"
Sep 11 17:48:03.547834 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.547749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 17:48:03.573109 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.573076    2563 generic.go:356] "Generic (PLEG): container finished" podID="769ab66a-cade-43e5-a337-c6697aaa6aca" containerID="3fbfcda9bddbb38f0b2ccb090793fc2cb59a6426cc319021d549583aa7210df8" exitCode=0
Sep 11 17:48:03.573282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.573150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" event={"ID":"769ab66a-cade-43e5-a337-c6697aaa6aca","Type":"ContainerDied","Data":"3fbfcda9bddbb38f0b2ccb090793fc2cb59a6426cc319021d549583aa7210df8"}
Sep 11 17:48:03.573583 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.573561    2563 scope.go:117] "RemoveContainer" containerID="3fbfcda9bddbb38f0b2ccb090793fc2cb59a6426cc319021d549583aa7210df8"
Sep 11 17:48:03.574604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.574581    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:48:03.574661 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.574645    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:48:03.574697 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:03.574676    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:48:04.583403 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:04.583358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" event={"ID":"769ab66a-cade-43e5-a337-c6697aaa6aca","Type":"ContainerStarted","Data":"75b1d319782900efcbeff757d50ec26a4a381531b9d5072edba58148895a8a4c"}
Sep 11 17:48:04.910670 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:04.910567    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:48:04.910906 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:04.910879    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4grfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 17:48:04.911001 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:04.910954    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" podUID="769ab66a-cade-43e5-a337-c6697aaa6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 17:48:05.910761 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:05.910728    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4grfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 17:48:05.911156 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:05.910795    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" podUID="769ab66a-cade-43e5-a337-c6697aaa6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 17:48:06.910936 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:06.910902    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4grfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 17:48:06.911418 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:06.910964    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" podUID="769ab66a-cade-43e5-a337-c6697aaa6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 17:48:07.910832 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:07.910794    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4grfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 17:48:07.911023 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:07.910868    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" podUID="769ab66a-cade-43e5-a337-c6697aaa6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 17:48:08.911120 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:08.911079    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4grfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 11 17:48:08.911642 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:08.911150    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw" podUID="769ab66a-cade-43e5-a337-c6697aaa6aca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 11 17:48:09.914557 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:09.914525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:48:09.914978 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:09.914826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:48:09.915606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:09.915586    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4grfw"
Sep 11 17:48:10.565468 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.565403    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"]
Sep 11 17:48:10.568192 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.568171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.574565 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.571942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 17:48:10.574565 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.572454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 17:48:10.574565 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.572829    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xdfcz\""
Sep 11 17:48:10.574565 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.573071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 17:48:10.582376 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.582344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"]
Sep 11 17:48:10.584158 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.584121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.584283 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.584199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.584327 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.584295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.584374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.584336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.584374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.584364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n885b\" (UniqueName: \"kubernetes.io/projected/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-kube-api-access-n885b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.584450 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.584423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685091 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n885b\" (UniqueName: \"kubernetes.io/projected/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-kube-api-access-n885b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685686 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.685766 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.685746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.687674 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.687642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.687972 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.687953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.695101 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.695076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n885b\" (UniqueName: \"kubernetes.io/projected/7fb349bc-ceac-461d-bbe7-8f1cf719a69e-kube-api-access-n885b\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m\" (UID: \"7fb349bc-ceac-461d-bbe7-8f1cf719a69e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:10.885132 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:10.885091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:11.165640 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:11.165610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"]
Sep 11 17:48:11.167190 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:48:11.167161    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb349bc_ceac_461d_bbe7_8f1cf719a69e.slice/crio-cf43328692dbb6e54dbfeb4c22c0ec25d293dda2064c7c6c15ef71073d285c24 WatchSource:0}: Error finding container cf43328692dbb6e54dbfeb4c22c0ec25d293dda2064c7c6c15ef71073d285c24: Status 404 returned error can't find the container with id cf43328692dbb6e54dbfeb4c22c0ec25d293dda2064c7c6c15ef71073d285c24
Sep 11 17:48:11.636005 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:11.635963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m" event={"ID":"7fb349bc-ceac-461d-bbe7-8f1cf719a69e","Type":"ContainerStarted","Data":"cf43328692dbb6e54dbfeb4c22c0ec25d293dda2064c7c6c15ef71073d285c24"}
Sep 11 17:48:12.643428 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:12.643383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m" event={"ID":"7fb349bc-ceac-461d-bbe7-8f1cf719a69e","Type":"ContainerStarted","Data":"a3ae63c79d00912c3986190eee9d61a92fccafa7acd361ab0f6671571d356d01"}
Sep 11 17:48:12.666241 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:12.666187    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m" podStartSLOduration=1.5293524939999998 podStartE2EDuration="2.666170703s" totalImagesPullingTime="1.136818209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:12.664853683 +0000 UTC m=+559.428308402" watchObservedRunningTime="2026-09-11 17:48:12.666170703 +0000 UTC m=+559.429625429"
Sep 11 17:48:20.885992 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:20.885953    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:20.886388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:20.886024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:20.898617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:20.898581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:21.704123 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:21.704080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m"
Sep 11 17:48:30.806929 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.806888    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"]
Sep 11 17:48:30.809480 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.809458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.812518 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.812499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 17:48:30.822997 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.822972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"]
Sep 11 17:48:30.843753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.843725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.843753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.843762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/380bf1fc-44af-4dfe-9f89-35155776e086-kube-api-access-rsstq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.843986 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.843900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380bf1fc-44af-4dfe-9f89-35155776e086-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.843986 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.843945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.843986 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.843967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.844093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.844010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.944797 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.944761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.944797 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.944803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.944825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/380bf1fc-44af-4dfe-9f89-35155776e086-kube-api-access-rsstq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.944893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380bf1fc-44af-4dfe-9f89-35155776e086-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.944915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.944938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.945164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.945249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.945365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.945293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.947922 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.947895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/380bf1fc-44af-4dfe-9f89-35155776e086-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.948268 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.948240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/380bf1fc-44af-4dfe-9f89-35155776e086-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:30.955212 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:30.955183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsstq\" (UniqueName: \"kubernetes.io/projected/380bf1fc-44af-4dfe-9f89-35155776e086-kube-api-access-rsstq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cbg8f\" (UID: \"380bf1fc-44af-4dfe-9f89-35155776e086\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:31.119637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:31.119586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:31.400825 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:31.400718    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"]
Sep 11 17:48:31.403022 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:48:31.402996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380bf1fc_44af_4dfe_9f89_35155776e086.slice/crio-90fb6252579e5b427e958629090e7643d097e347d7a0d12d0c7d138a8845423b WatchSource:0}: Error finding container 90fb6252579e5b427e958629090e7643d097e347d7a0d12d0c7d138a8845423b: Status 404 returned error can't find the container with id 90fb6252579e5b427e958629090e7643d097e347d7a0d12d0c7d138a8845423b
Sep 11 17:48:31.747957 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:31.747867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f" event={"ID":"380bf1fc-44af-4dfe-9f89-35155776e086","Type":"ContainerStarted","Data":"0b836dd35bb41eb57530a7624dad21456bca22931eb9c083d549bd8504fc6a04"}
Sep 11 17:48:31.747957 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:31.747921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f" event={"ID":"380bf1fc-44af-4dfe-9f89-35155776e086","Type":"ContainerStarted","Data":"90fb6252579e5b427e958629090e7643d097e347d7a0d12d0c7d138a8845423b"}
Sep 11 17:48:31.769336 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:31.769181    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f" podStartSLOduration=1.5980864590000001 podStartE2EDuration="1.769159076s" totalImagesPullingTime="171.072617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:31.768935062 +0000 UTC m=+578.532389790" watchObservedRunningTime="2026-09-11 17:48:31.769159076 +0000 UTC m=+578.532613803"
Sep 11 17:48:41.120595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:41.120555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:41.121208 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:41.120628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:41.140149 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:41.140118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:41.815716 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:41.815685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cbg8f"
Sep 11 17:48:53.703511 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:53.703469    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:48:53.704016 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:48:53.703591    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:50:59.582754 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.582675    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"]
Sep 11 17:50:59.588880 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.588849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.595411 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.595386    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-715253-openshift-default-dockercfg-nbnpr\""
Sep 11 17:50:59.610704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.610676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"]
Sep 11 17:50:59.692094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.692061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"]
Sep 11 17:50:59.695857 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.695836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.696204 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-envoy\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1af3b5-3380-46da-b3f4-d332e02c61da-istiod-ca-cert\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696282 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvbt\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-kube-api-access-9qvbt\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696377 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-data\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696377 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-socket\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696457 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-credential-socket\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696526 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-certs\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696634 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-podinfo\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.696681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.696661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-token\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.699824 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.699801    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-286c27-openshift-default-dockercfg-xqzsg\""
Sep 11 17:50:59.717590 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.717562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"]
Sep 11 17:50:59.797432 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-podinfo\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797432 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-data\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.797706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-token\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1af3b5-3380-46da-b3f4-d332e02c61da-istiod-ca-cert\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-data\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-envoy\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.797706 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-socket\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-credential-socket\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-certs\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7dw\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-kube-api-access-4w7dw\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.797962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-envoy\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.797962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvbt\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-kube-api-access-9qvbt\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.797964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-data\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a9a098-d486-4015-9253-107d0a3ab937-istiod-ca-cert\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-socket\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-credential-socket\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a9a098-d486-4015-9253-107d0a3ab937-istio-podinfo\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-istio-token\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-socket\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-certs\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-credential-socket\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-certs\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.798556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.798452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1af3b5-3380-46da-b3f4-d332e02c61da-istiod-ca-cert\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.800054 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.800030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-podinfo\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.800302 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.800284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-envoy\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.827910 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.827875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-token\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.828091 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.828045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvbt\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-kube-api-access-9qvbt\") pod \"e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") " pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.899429 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-socket\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899429 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-certs\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-data\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-envoy\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7dw\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-kube-api-access-4w7dw\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899704 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a9a098-d486-4015-9253-107d0a3ab937-istiod-ca-cert\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899949 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-credential-socket\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899949 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:50:59.899949 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a9a098-d486-4015-9253-107d0a3ab937-istio-podinfo\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899949 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-istio-token\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899949 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-socket\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.899949 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-certs\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.900332 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.899994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-data\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.900332 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.900248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-credential-socket\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.900753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.900732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a9a098-d486-4015-9253-107d0a3ab937-istiod-ca-cert\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.902322 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.902299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-envoy\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.902585 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.902567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a9a098-d486-4015-9253-107d0a3ab937-istio-podinfo\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.912314 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.912286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7dw\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-kube-api-access-4w7dw\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:50:59.913344 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:50:59.913326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-istio-token\") pod \"e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") " pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:00.007400 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.007359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:00.181110 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.181080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"]
Sep 11 17:51:00.182359 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:51:00.182327    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1af3b5_3380_46da_b3f4_d332e02c61da.slice/crio-25bd4adc4284fdcb182b8e8602debdf03b1c01d311f9b22447c0313278be6711 WatchSource:0}: Error finding container 25bd4adc4284fdcb182b8e8602debdf03b1c01d311f9b22447c0313278be6711: Status 404 returned error can't find the container with id 25bd4adc4284fdcb182b8e8602debdf03b1c01d311f9b22447c0313278be6711
Sep 11 17:51:00.184696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.184666    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:00.184790 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.184729    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:00.184790 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.184755    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:00.289802 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.289774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"]
Sep 11 17:51:00.292295 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:51:00.292252    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a9a098_d486_4015_9253_107d0a3ab937.slice/crio-eedf746a1276b986f19d61414bc88f9a5595f3bce5a820a28963e641b5bd36a2 WatchSource:0}: Error finding container eedf746a1276b986f19d61414bc88f9a5595f3bce5a820a28963e641b5bd36a2: Status 404 returned error can't find the container with id eedf746a1276b986f19d61414bc88f9a5595f3bce5a820a28963e641b5bd36a2
Sep 11 17:51:00.294810 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.294778    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:00.294885 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.294847    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:00.294885 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.294873    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:00.553535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.553402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" event={"ID":"cd1af3b5-3380-46da-b3f4-d332e02c61da","Type":"ContainerStarted","Data":"be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"}
Sep 11 17:51:00.553535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.553473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" event={"ID":"cd1af3b5-3380-46da-b3f4-d332e02c61da","Type":"ContainerStarted","Data":"25bd4adc4284fdcb182b8e8602debdf03b1c01d311f9b22447c0313278be6711"}
Sep 11 17:51:00.555290 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.555246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" event={"ID":"97a9a098-d486-4015-9253-107d0a3ab937","Type":"ContainerStarted","Data":"0a0b077d808911a146c4b12b1b20b739fdfd8932575b89dbd72c332e4f3b857e"}
Sep 11 17:51:00.555434 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.555297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" event={"ID":"97a9a098-d486-4015-9253-107d0a3ab937","Type":"ContainerStarted","Data":"eedf746a1276b986f19d61414bc88f9a5595f3bce5a820a28963e641b5bd36a2"}
Sep 11 17:51:00.579746 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.579678    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podStartSLOduration=1.579658507 podStartE2EDuration="1.579658507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:00.575936457 +0000 UTC m=+727.339391222" watchObservedRunningTime="2026-09-11 17:51:00.579658507 +0000 UTC m=+727.343113236"
Sep 11 17:51:00.608258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.608197    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podStartSLOduration=1.6081730379999999 podStartE2EDuration="1.608173038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:00.606103798 +0000 UTC m=+727.369558524" watchObservedRunningTime="2026-09-11 17:51:00.608173038 +0000 UTC m=+727.371627768"
Sep 11 17:51:00.900685 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:00.900649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:01.007644 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:01.007600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:01.009194 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:01.009170    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:01.009277 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:01.009220    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:01.906189 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:01.906155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:02.012560 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:02.012525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:02.567388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:02.567348    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:02.567596 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:02.567408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:02.568368 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:02.568344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:02.568472 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:02.568406    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:03.995478 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:03.995434    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"]
Sep 11 17:51:03.998940 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:03.998920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.002573 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.002550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-286c27-serving-cert\""
Sep 11 17:51:04.039134 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.039099    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"]
Sep 11 17:51:04.041324 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.041293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-metrics-certs\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.041471 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.041338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bph\" (UniqueName: \"kubernetes.io/projected/f60cfefc-2c06-4d92-9e4e-42aff11ef803-kube-api-access-59bph\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.041471 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.041446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-maas-api-tls\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.041589 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.041477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60cfefc-2c06-4d92-9e4e-42aff11ef803-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.142270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.142239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-maas-api-tls\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.142270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.142277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60cfefc-2c06-4d92-9e4e-42aff11ef803-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.142584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.142331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-metrics-certs\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.142584 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.142350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bph\" (UniqueName: \"kubernetes.io/projected/f60cfefc-2c06-4d92-9e4e-42aff11ef803-kube-api-access-59bph\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.143157 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.143080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60cfefc-2c06-4d92-9e4e-42aff11ef803-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.145013 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.144986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-metrics-certs\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.145106 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.145017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-maas-api-tls\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.170188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.170150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bph\" (UniqueName: \"kubernetes.io/projected/f60cfefc-2c06-4d92-9e4e-42aff11ef803-kube-api-access-59bph\") pod \"maas-api-e2e-worker-w0-286c27-84c79b57b-8m879\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.309799 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.309713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:04.597593 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.597560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"]
Sep 11 17:51:04.599815 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:51:04.599784    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60cfefc_2c06_4d92_9e4e_42aff11ef803.slice/crio-2f416da83add4d0d314d49946ea3bcccd0cd59fd5708d597225cb1eb36ddb846 WatchSource:0}: Error finding container 2f416da83add4d0d314d49946ea3bcccd0cd59fd5708d597225cb1eb36ddb846: Status 404 returned error can't find the container with id 2f416da83add4d0d314d49946ea3bcccd0cd59fd5708d597225cb1eb36ddb846
Sep 11 17:51:04.601705 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:04.601680    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:51:05.447231 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.447131    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"]
Sep 11 17:51:05.451612 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.451581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.457709 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.457687    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-715253-serving-cert\""
Sep 11 17:51:05.476809 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.476779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"]
Sep 11 17:51:05.557883 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.557849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.557883 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.557885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527t9\" (UniqueName: \"kubernetes.io/projected/d1649022-7039-4c08-8dcd-a2b9b3a169d5-kube-api-access-527t9\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.558086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.557920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-metrics-certs\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.558086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.558001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1649022-7039-4c08-8dcd-a2b9b3a169d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.589535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.589466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879" event={"ID":"f60cfefc-2c06-4d92-9e4e-42aff11ef803","Type":"ContainerStarted","Data":"96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a"}
Sep 11 17:51:05.589727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.589551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879" event={"ID":"f60cfefc-2c06-4d92-9e4e-42aff11ef803","Type":"ContainerStarted","Data":"2f416da83add4d0d314d49946ea3bcccd0cd59fd5708d597225cb1eb36ddb846"}
Sep 11 17:51:05.589727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.589606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:05.618454 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.618399    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879" podStartSLOduration=2.130626816 podStartE2EDuration="2.618382529s" totalImagesPullingTime="487.755713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:05.616280139 +0000 UTC m=+732.379734866" watchObservedRunningTime="2026-09-11 17:51:05.618382529 +0000 UTC m=+732.381837256"
Sep 11 17:51:05.658820 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.658776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.659029 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.658828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527t9\" (UniqueName: \"kubernetes.io/projected/d1649022-7039-4c08-8dcd-a2b9b3a169d5-kube-api-access-527t9\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.659029 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.658879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-metrics-certs\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.659029 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.658911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1649022-7039-4c08-8dcd-a2b9b3a169d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.659209 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:51:05.658881    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-serving-cert: secret "maas-api-e2e-inf-a-715253-serving-cert" not found
Sep 11 17:51:05.659209 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:51:05.659102    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls podName:d1649022-7039-4c08-8dcd-a2b9b3a169d5 nodeName:}" failed. No retries permitted until 2026-09-11 17:51:06.159078362 +0000 UTC m=+732.922533067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls") pod "maas-api-e2e-inf-a-715253-7656949976-g87jc" (UID: "d1649022-7039-4c08-8dcd-a2b9b3a169d5") : secret "maas-api-e2e-inf-a-715253-serving-cert" not found
Sep 11 17:51:05.659700 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.659673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1649022-7039-4c08-8dcd-a2b9b3a169d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.661579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.661559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-metrics-certs\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:05.671333 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:05.671305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-527t9\" (UniqueName: \"kubernetes.io/projected/d1649022-7039-4c08-8dcd-a2b9b3a169d5-kube-api-access-527t9\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:06.164986 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:06.164949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:06.167788 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:06.167759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-715253-7656949976-g87jc\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:06.363012 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:06.362973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:06.648030 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:06.647975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"]
Sep 11 17:51:07.613874 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:07.613837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc" event={"ID":"d1649022-7039-4c08-8dcd-a2b9b3a169d5","Type":"ContainerStarted","Data":"baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1"}
Sep 11 17:51:07.614058 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:07.613897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc" event={"ID":"d1649022-7039-4c08-8dcd-a2b9b3a169d5","Type":"ContainerStarted","Data":"c4657c756ff38dbad53507a67338d42e489a35d8fbb233ba5d2eecec5bf9285f"}
Sep 11 17:51:07.614058 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:07.613924    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:07.636619 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:07.636560    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc" podStartSLOduration=2.281078064 podStartE2EDuration="2.636538909s" totalImagesPullingTime="355.460845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:07.634715029 +0000 UTC m=+734.398169757" watchObservedRunningTime="2026-09-11 17:51:07.636538909 +0000 UTC m=+734.399993636"
Sep 11 17:51:09.678543 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.678508    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"]
Sep 11 17:51:09.685582 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.685526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.689140 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.689112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4ab253-serving-cert\""
Sep 11 17:51:09.706648 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.706619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"]
Sep 11 17:51:09.803108 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.803074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.803284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.803121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6845c5ec-5000-4613-8edc-d0d8d165027e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.803284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.803191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.803284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.803280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7v2x\" (UniqueName: \"kubernetes.io/projected/6845c5ec-5000-4613-8edc-d0d8d165027e-kube-api-access-z7v2x\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.904106 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.904066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.904312 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.904156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7v2x\" (UniqueName: \"kubernetes.io/projected/6845c5ec-5000-4613-8edc-d0d8d165027e-kube-api-access-z7v2x\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.904312 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:51:09.904233    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-serving-cert: secret "maas-api-e2e-worker-w4-4ab253-serving-cert" not found
Sep 11 17:51:09.904312 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:51:09.904305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls podName:6845c5ec-5000-4613-8edc-d0d8d165027e nodeName:}" failed. No retries permitted until 2026-09-11 17:51:10.40428393 +0000 UTC m=+737.167738639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls") pod "maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" (UID: "6845c5ec-5000-4613-8edc-d0d8d165027e") : secret "maas-api-e2e-worker-w4-4ab253-serving-cert" not found
Sep 11 17:51:09.904312 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.904235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.904589 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.904364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6845c5ec-5000-4613-8edc-d0d8d165027e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.905168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.905145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6845c5ec-5000-4613-8edc-d0d8d165027e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.907140 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.907119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-metrics-certs\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:09.918823 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:09.918797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7v2x\" (UniqueName: \"kubernetes.io/projected/6845c5ec-5000-4613-8edc-d0d8d165027e-kube-api-access-z7v2x\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:10.410816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:10.410777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:10.413530 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:10.413483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:10.612719 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:10.612673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:10.892254 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:10.892224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"]
Sep 11 17:51:10.894577 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:51:10.894547    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6845c5ec_5000_4613_8edc_d0d8d165027e.slice/crio-6a2ce31b3385023f9445745bc7df66813efc4f33333fe0a2f924160f7cbde634 WatchSource:0}: Error finding container 6a2ce31b3385023f9445745bc7df66813efc4f33333fe0a2f924160f7cbde634: Status 404 returned error can't find the container with id 6a2ce31b3385023f9445745bc7df66813efc4f33333fe0a2f924160f7cbde634
Sep 11 17:51:11.606981 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:11.606880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 17:51:11.644750 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:11.644699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" event={"ID":"6845c5ec-5000-4613-8edc-d0d8d165027e","Type":"ContainerStarted","Data":"b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84"}
Sep 11 17:51:11.644897 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:11.644780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" event={"ID":"6845c5ec-5000-4613-8edc-d0d8d165027e","Type":"ContainerStarted","Data":"6a2ce31b3385023f9445745bc7df66813efc4f33333fe0a2f924160f7cbde634"}
Sep 11 17:51:11.644975 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:11.644914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:11.675015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:11.674959    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" podStartSLOduration=2.242550004 podStartE2EDuration="2.674941767s" totalImagesPullingTime="432.391763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:11.674295907 +0000 UTC m=+738.437750633" watchObservedRunningTime="2026-09-11 17:51:11.674941767 +0000 UTC m=+738.438396494"
Sep 11 17:51:13.644912 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:13.644872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:51:17.663047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:17.662962    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 17:51:18.608718 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:18.608660    2563 patch_prober.go:28] interesting pod/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 17:51:18.608950 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:18.608739    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 17:51:19.662838 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.662795    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"]
Sep 11 17:51:19.667239 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.667217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.670462 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.670440    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-37899e-dockercfg-v8rbj\""
Sep 11 17:51:19.671147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.671132    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-37899e\""
Sep 11 17:51:19.678191 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.678162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"]
Sep 11 17:51:19.814887 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.814851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72437013-8ec6-4dcb-88f4-06f601ea8608-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.815075 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.814908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7npf\" (UniqueName: \"kubernetes.io/projected/72437013-8ec6-4dcb-88f4-06f601ea8608-kube-api-access-b7npf\") pod \"payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.916316 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.916211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72437013-8ec6-4dcb-88f4-06f601ea8608-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.916482 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.916399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7npf\" (UniqueName: \"kubernetes.io/projected/72437013-8ec6-4dcb-88f4-06f601ea8608-kube-api-access-b7npf\") pod \"payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.917138 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.917113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72437013-8ec6-4dcb-88f4-06f601ea8608-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.937241 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.937207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7npf\" (UniqueName: \"kubernetes.io/projected/72437013-8ec6-4dcb-88f4-06f601ea8608-kube-api-access-b7npf\") pod \"payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:19.946824 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.946788    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"]
Sep 11 17:51:19.951389 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.951361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:19.955170 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.955143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-50ce3a65-serving-cert\""
Sep 11 17:51:19.969204 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.969170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"]
Sep 11 17:51:19.981436 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:19.981399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:20.118752 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.118716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczfr\" (UniqueName: \"kubernetes.io/projected/e366aebf-193f-401d-a364-991631755139-kube-api-access-tczfr\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.118947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.118762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e366aebf-193f-401d-a364-991631755139-openshift-service-ca\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.118947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.118873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-maas-api-tls\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.118947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.118929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-metrics-certs\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.220681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.220585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczfr\" (UniqueName: \"kubernetes.io/projected/e366aebf-193f-401d-a364-991631755139-kube-api-access-tczfr\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.220681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.220631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e366aebf-193f-401d-a364-991631755139-openshift-service-ca\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.220681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.220669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-maas-api-tls\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.220979 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.220688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-metrics-certs\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.221538 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.221457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e366aebf-193f-401d-a364-991631755139-openshift-service-ca\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.223944 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.223896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-metrics-certs\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.224579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.224556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-maas-api-tls\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.230566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.230533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczfr\" (UniqueName: \"kubernetes.io/projected/e366aebf-193f-401d-a364-991631755139-kube-api-access-tczfr\") pod \"maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl\" (UID: \"e366aebf-193f-401d-a364-991631755139\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.264726 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.264689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:20.278649 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.278617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"]
Sep 11 17:51:20.560594 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.560558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"]
Sep 11 17:51:20.561477 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:51:20.561437    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode366aebf_193f_401d_a364_991631755139.slice/crio-437576207fed56b312417d5b6fed2e418b5cb1e86a5c2af1ca1c747bed3b08c0 WatchSource:0}: Error finding container 437576207fed56b312417d5b6fed2e418b5cb1e86a5c2af1ca1c747bed3b08c0: Status 404 returned error can't find the container with id 437576207fed56b312417d5b6fed2e418b5cb1e86a5c2af1ca1c747bed3b08c0
Sep 11 17:51:20.723724 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.723607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl" event={"ID":"e366aebf-193f-401d-a364-991631755139","Type":"ContainerStarted","Data":"437576207fed56b312417d5b6fed2e418b5cb1e86a5c2af1ca1c747bed3b08c0"}
Sep 11 17:51:20.725337 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:20.725309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl" event={"ID":"72437013-8ec6-4dcb-88f4-06f601ea8608","Type":"ContainerStarted","Data":"e16f554c6982a2979b98cd8b613c817eabe67fc7a6dbeed4665f3c6c0dc42638"}
Sep 11 17:51:21.735022 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.734946    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerID="be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741" exitCode=0
Sep 11 17:51:21.735022 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.734991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" event={"ID":"cd1af3b5-3380-46da-b3f4-d332e02c61da","Type":"ContainerDied","Data":"be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"}
Sep 11 17:51:21.735606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.735535    2563 scope.go:117] "RemoveContainer" containerID="be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"
Sep 11 17:51:21.737253 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.737225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl" event={"ID":"e366aebf-193f-401d-a364-991631755139","Type":"ContainerStarted","Data":"9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f"}
Sep 11 17:51:21.737385 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.737322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:21.739012 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.738989    2563 generic.go:356] "Generic (PLEG): container finished" podID="97a9a098-d486-4015-9253-107d0a3ab937" containerID="0a0b077d808911a146c4b12b1b20b739fdfd8932575b89dbd72c332e4f3b857e" exitCode=0
Sep 11 17:51:21.739109 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.739056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" event={"ID":"97a9a098-d486-4015-9253-107d0a3ab937","Type":"ContainerDied","Data":"0a0b077d808911a146c4b12b1b20b739fdfd8932575b89dbd72c332e4f3b857e"}
Sep 11 17:51:21.739477 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.739452    2563 scope.go:117] "RemoveContainer" containerID="0a0b077d808911a146c4b12b1b20b739fdfd8932575b89dbd72c332e4f3b857e"
Sep 11 17:51:21.813364 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:21.813246    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl" podStartSLOduration=2.446219472 podStartE2EDuration="2.813221279s" totalImagesPullingTime="367.001807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:21.809040861 +0000 UTC m=+748.572495588" watchObservedRunningTime="2026-09-11 17:51:21.813221279 +0000 UTC m=+748.576676007"
Sep 11 17:51:22.008678 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.008629    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:22.008678 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.008661    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:22.008869 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.008707    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:22.008869 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.008728    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:22.008869 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.008739    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:22.008869 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.008762    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:51:22.755310 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.755165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" event={"ID":"cd1af3b5-3380-46da-b3f4-d332e02c61da","Type":"ContainerStarted","Data":"66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a"}
Sep 11 17:51:22.757578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.757535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" event={"ID":"97a9a098-d486-4015-9253-107d0a3ab937","Type":"ContainerStarted","Data":"783ca1e75c45cc7d0202950fb22e14d0b57bd6d0738b50648d3a14563f6e9904"}
Sep 11 17:51:22.759354 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.759325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl" event={"ID":"72437013-8ec6-4dcb-88f4-06f601ea8608","Type":"ContainerStarted","Data":"b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1"}
Sep 11 17:51:22.759523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.759458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:22.805238 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.805169    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl" podStartSLOduration=2.036781032 podStartE2EDuration="3.805143118s" totalImagesPullingTime="1.768362086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:22.802307794 +0000 UTC m=+749.565762521" watchObservedRunningTime="2026-09-11 17:51:22.805143118 +0000 UTC m=+749.568597847"
Sep 11 17:51:22.900428 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:22.900391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:23.008837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:23.008751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:23.009095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:23.008808    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:23.009331 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:23.009304    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:23.900769 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:23.900727    2563 patch_prober.go:28] interesting pod/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:51:23.901355 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:23.901323    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 17:51:24.007884 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:24.007838    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:24.008099 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:24.007910    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:24.901271 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:24.901231    2563 patch_prober.go:28] interesting pod/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:51:24.901902 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:24.901865    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 17:51:25.007888 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:25.007851    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:25.008241 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:25.008200    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:25.901128 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:25.901078    2563 patch_prober.go:28] interesting pod/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:51:25.901356 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:25.901151    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 17:51:26.008611 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:26.008132    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:26.008611 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:26.008200    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:26.900904 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:26.900854    2563 patch_prober.go:28] interesting pod/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:51:26.901126 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:26.900949    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 17:51:27.009055 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:27.009015    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:27.009674 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:27.009612    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:27.765866 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:27.765833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:27.904518 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:27.903909    2563 patch_prober.go:28] interesting pod/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:51:27.904518 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:27.904000    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 11 17:51:28.008735 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.008690    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 11 17:51:28.008919 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.008778    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 11 17:51:28.906031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.906002    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:28.906463 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.906322    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:28.907200 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.907178    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:51:28.949976 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.949936    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"]
Sep 11 17:51:28.953275 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.953248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:28.956555 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.956521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9aab53-serving-cert\""
Sep 11 17:51:28.961593 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:28.961569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"]
Sep 11 17:51:29.044708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.044381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:29.044911 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.044734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.044911 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.044849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ee9138-0121-4b07-a24f-2b86c4ea10af-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.045035 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.044948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-metrics-certs\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.045035 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.045011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6f9t\" (UniqueName: \"kubernetes.io/projected/a4ee9138-0121-4b07-a24f-2b86c4ea10af-kube-api-access-q6f9t\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.045287 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.045226    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:29.046443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.046414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 17:51:29.146173 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.146137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ee9138-0121-4b07-a24f-2b86c4ea10af-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.146334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.146253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-metrics-certs\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.146481 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.146459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6f9t\" (UniqueName: \"kubernetes.io/projected/a4ee9138-0121-4b07-a24f-2b86c4ea10af-kube-api-access-q6f9t\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.146578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.146533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.146907 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.146842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ee9138-0121-4b07-a24f-2b86c4ea10af-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.148952 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.148931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-maas-api-tls\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.149285 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.149265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-metrics-certs\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.155213 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.155186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6f9t\" (UniqueName: \"kubernetes.io/projected/a4ee9138-0121-4b07-a24f-2b86c4ea10af-kube-api-access-q6f9t\") pod \"maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.270913 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.270819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:29.553818 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.553795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"]
Sep 11 17:51:29.556015 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:51:29.555977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ee9138_0121_4b07_a24f_2b86c4ea10af.slice/crio-d2b675f54c326fb700c012c27fcc302166749c1185cb017f3b8e37a6bbcad506 WatchSource:0}: Error finding container d2b675f54c326fb700c012c27fcc302166749c1185cb017f3b8e37a6bbcad506: Status 404 returned error can't find the container with id d2b675f54c326fb700c012c27fcc302166749c1185cb017f3b8e37a6bbcad506
Sep 11 17:51:29.866244 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:29.866210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p" event={"ID":"a4ee9138-0121-4b07-a24f-2b86c4ea10af","Type":"ContainerStarted","Data":"d2b675f54c326fb700c012c27fcc302166749c1185cb017f3b8e37a6bbcad506"}
Sep 11 17:51:30.874754 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:30.874719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p" event={"ID":"a4ee9138-0121-4b07-a24f-2b86c4ea10af","Type":"ContainerStarted","Data":"7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075"}
Sep 11 17:51:30.875128 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:30.874942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:30.897057 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:30.896995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p" podStartSLOduration=2.4379771789999998 podStartE2EDuration="2.89697282s" totalImagesPullingTime="458.995641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:51:30.893517659 +0000 UTC m=+757.656972387" watchObservedRunningTime="2026-09-11 17:51:30.89697282 +0000 UTC m=+757.660427548"
Sep 11 17:51:33.767879 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:33.767844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:51:36.885714 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:36.885686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:51:56.469821 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.469775    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"]
Sep 11 17:51:56.470408 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.469989    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl" podUID="e366aebf-193f-401d-a364-991631755139" containerName="maas-api" containerID="cri-o://9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f" gracePeriod=30
Sep 11 17:51:56.728376 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.728305    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:56.824953 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.824918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tczfr\" (UniqueName: \"kubernetes.io/projected/e366aebf-193f-401d-a364-991631755139-kube-api-access-tczfr\") pod \"e366aebf-193f-401d-a364-991631755139\" (UID: \"e366aebf-193f-401d-a364-991631755139\") "
Sep 11 17:51:56.825144 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.825022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-maas-api-tls\") pod \"e366aebf-193f-401d-a364-991631755139\" (UID: \"e366aebf-193f-401d-a364-991631755139\") "
Sep 11 17:51:56.825144 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.825071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e366aebf-193f-401d-a364-991631755139-openshift-service-ca\") pod \"e366aebf-193f-401d-a364-991631755139\" (UID: \"e366aebf-193f-401d-a364-991631755139\") "
Sep 11 17:51:56.825688 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.825165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-metrics-certs\") pod \"e366aebf-193f-401d-a364-991631755139\" (UID: \"e366aebf-193f-401d-a364-991631755139\") "
Sep 11 17:51:56.825688 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.825571    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e366aebf-193f-401d-a364-991631755139-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e366aebf-193f-401d-a364-991631755139" (UID: "e366aebf-193f-401d-a364-991631755139"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:51:56.828401 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.828337    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e366aebf-193f-401d-a364-991631755139" (UID: "e366aebf-193f-401d-a364-991631755139"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:51:56.828726 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.828347    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e366aebf-193f-401d-a364-991631755139" (UID: "e366aebf-193f-401d-a364-991631755139"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:51:56.829431 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.829393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e366aebf-193f-401d-a364-991631755139-kube-api-access-tczfr" (OuterVolumeSpecName: "kube-api-access-tczfr") pod "e366aebf-193f-401d-a364-991631755139" (UID: "e366aebf-193f-401d-a364-991631755139"). InnerVolumeSpecName "kube-api-access-tczfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:51:56.926608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.926569    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:51:56.926608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.926603    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tczfr\" (UniqueName: \"kubernetes.io/projected/e366aebf-193f-401d-a364-991631755139-kube-api-access-tczfr\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:51:56.926608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.926615    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e366aebf-193f-401d-a364-991631755139-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:51:56.926854 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:56.926623    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e366aebf-193f-401d-a364-991631755139-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:51:57.037085 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.036997    2563 generic.go:356] "Generic (PLEG): container finished" podID="e366aebf-193f-401d-a364-991631755139" containerID="9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f" exitCode=0
Sep 11 17:51:57.037085 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.037046    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"
Sep 11 17:51:57.037334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.037080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl" event={"ID":"e366aebf-193f-401d-a364-991631755139","Type":"ContainerDied","Data":"9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f"}
Sep 11 17:51:57.037334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.037141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl" event={"ID":"e366aebf-193f-401d-a364-991631755139","Type":"ContainerDied","Data":"437576207fed56b312417d5b6fed2e418b5cb1e86a5c2af1ca1c747bed3b08c0"}
Sep 11 17:51:57.037334 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.037175    2563 scope.go:117] "RemoveContainer" containerID="9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f"
Sep 11 17:51:57.047632 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.047616    2563 scope.go:117] "RemoveContainer" containerID="9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f"
Sep 11 17:51:57.047941 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:51:57.047921    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f\": container with ID starting with 9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f not found: ID does not exist" containerID="9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f"
Sep 11 17:51:57.047997 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.047951    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f"} err="failed to get container status \"9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f\": rpc error: code = NotFound desc = could not find container \"9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f\": container with ID starting with 9668bdda8845e9a7907d3b4d7193aa742ccb83c330f31ed39ea6ee7b833b1f3f not found: ID does not exist"
Sep 11 17:51:57.061964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.061936    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"]
Sep 11 17:51:57.066798 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.066770    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50ce3a65-64ccfd6b86-4c5wl"]
Sep 11 17:51:57.781813 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:51:57.781777    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e366aebf-193f-401d-a364-991631755139" path="/var/lib/kubelet/pods/e366aebf-193f-401d-a364-991631755139/volumes"
Sep 11 17:52:03.459283 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.459239    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"]
Sep 11 17:52:03.460637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.460608    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e366aebf-193f-401d-a364-991631755139" containerName="maas-api"
Sep 11 17:52:03.460815 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.460802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e366aebf-193f-401d-a364-991631755139" containerName="maas-api"
Sep 11 17:52:03.461067 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.461052    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e366aebf-193f-401d-a364-991631755139" containerName="maas-api"
Sep 11 17:52:03.466265 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.466234    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.471065 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.471029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3f5d8d77\""
Sep 11 17:52:03.481095 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.481060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"]
Sep 11 17:52:03.597202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.597162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261f1d5b-9bbf-437b-b110-ce09f23e400c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.597382 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.597289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxzb\" (UniqueName: \"kubernetes.io/projected/261f1d5b-9bbf-437b-b110-ce09f23e400c-kube-api-access-lkxzb\") pod \"payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.699309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.698764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261f1d5b-9bbf-437b-b110-ce09f23e400c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.699309 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.698879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxzb\" (UniqueName: \"kubernetes.io/projected/261f1d5b-9bbf-437b-b110-ce09f23e400c-kube-api-access-lkxzb\") pod \"payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.699771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.699731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261f1d5b-9bbf-437b-b110-ce09f23e400c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.709096 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.709068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxzb\" (UniqueName: \"kubernetes.io/projected/261f1d5b-9bbf-437b-b110-ce09f23e400c-kube-api-access-lkxzb\") pod \"payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") " pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:03.786122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:03.786015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:04.064730 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:04.064696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"]
Sep 11 17:52:04.066649 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:52:04.066614    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261f1d5b_9bbf_437b_b110_ce09f23e400c.slice/crio-e7dc623edf3141b11da44d91e0935fb7703052b8a1ef9459a9d76db69dd50bf0 WatchSource:0}: Error finding container e7dc623edf3141b11da44d91e0935fb7703052b8a1ef9459a9d76db69dd50bf0: Status 404 returned error can't find the container with id e7dc623edf3141b11da44d91e0935fb7703052b8a1ef9459a9d76db69dd50bf0
Sep 11 17:52:04.094037 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:04.094000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm" event={"ID":"261f1d5b-9bbf-437b-b110-ce09f23e400c","Type":"ContainerStarted","Data":"e7dc623edf3141b11da44d91e0935fb7703052b8a1ef9459a9d76db69dd50bf0"}
Sep 11 17:52:05.101075 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:05.101033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm" event={"ID":"261f1d5b-9bbf-437b-b110-ce09f23e400c","Type":"ContainerStarted","Data":"a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455"}
Sep 11 17:52:05.101535 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:05.101155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:05.122689 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:05.122632    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm" podStartSLOduration=2.122615587 podStartE2EDuration="2.122615587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:52:05.119530567 +0000 UTC m=+791.882985295" watchObservedRunningTime="2026-09-11 17:52:05.122615587 +0000 UTC m=+791.886070313"
Sep 11 17:52:16.109720 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:16.109634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:31.788622 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:31.788536    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"]
Sep 11 17:52:31.789042 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:31.788814    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm" podUID="261f1d5b-9bbf-437b-b110-ce09f23e400c" containerName="payload-processing" containerID="cri-o://a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455" gracePeriod=30
Sep 11 17:52:32.070677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.068171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:32.190343 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.190300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261f1d5b-9bbf-437b-b110-ce09f23e400c-plugins-config-volume\") pod \"261f1d5b-9bbf-437b-b110-ce09f23e400c\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") "
Sep 11 17:52:32.190555 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.190444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkxzb\" (UniqueName: \"kubernetes.io/projected/261f1d5b-9bbf-437b-b110-ce09f23e400c-kube-api-access-lkxzb\") pod \"261f1d5b-9bbf-437b-b110-ce09f23e400c\" (UID: \"261f1d5b-9bbf-437b-b110-ce09f23e400c\") "
Sep 11 17:52:32.190810 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.190783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261f1d5b-9bbf-437b-b110-ce09f23e400c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "261f1d5b-9bbf-437b-b110-ce09f23e400c" (UID: "261f1d5b-9bbf-437b-b110-ce09f23e400c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:52:32.192904 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.192873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261f1d5b-9bbf-437b-b110-ce09f23e400c-kube-api-access-lkxzb" (OuterVolumeSpecName: "kube-api-access-lkxzb") pod "261f1d5b-9bbf-437b-b110-ce09f23e400c" (UID: "261f1d5b-9bbf-437b-b110-ce09f23e400c"). InnerVolumeSpecName "kube-api-access-lkxzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:52:32.277008 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.276963    2563 generic.go:356] "Generic (PLEG): container finished" podID="261f1d5b-9bbf-437b-b110-ce09f23e400c" containerID="a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455" exitCode=0
Sep 11 17:52:32.277236 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.277034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"
Sep 11 17:52:32.277236 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.277052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm" event={"ID":"261f1d5b-9bbf-437b-b110-ce09f23e400c","Type":"ContainerDied","Data":"a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455"}
Sep 11 17:52:32.277236 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.277146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm" event={"ID":"261f1d5b-9bbf-437b-b110-ce09f23e400c","Type":"ContainerDied","Data":"e7dc623edf3141b11da44d91e0935fb7703052b8a1ef9459a9d76db69dd50bf0"}
Sep 11 17:52:32.277236 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.277177    2563 scope.go:117] "RemoveContainer" containerID="a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455"
Sep 11 17:52:32.288216 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.288193    2563 scope.go:117] "RemoveContainer" containerID="a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455"
Sep 11 17:52:32.288597 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:52:32.288573    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455\": container with ID starting with a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455 not found: ID does not exist" containerID="a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455"
Sep 11 17:52:32.288707 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.288606    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455"} err="failed to get container status \"a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455\": rpc error: code = NotFound desc = could not find container \"a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455\": container with ID starting with a2a917167bfc923396e85318a3de7803458dbfce8058355aaaff128984a28455 not found: ID does not exist"
Sep 11 17:52:32.291836 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.291810    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxzb\" (UniqueName: \"kubernetes.io/projected/261f1d5b-9bbf-437b-b110-ce09f23e400c-kube-api-access-lkxzb\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:52:32.291836 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.291835    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/261f1d5b-9bbf-437b-b110-ce09f23e400c-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:52:32.301654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.301575    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"]
Sep 11 17:52:32.308041 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.308016    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3f5d8d77-8c8877b46-j8rpm"]
Sep 11 17:52:32.961148 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.961116    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"]
Sep 11 17:52:32.961562 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.961554    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261f1d5b-9bbf-437b-b110-ce09f23e400c" containerName="payload-processing"
Sep 11 17:52:32.961611 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.961568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="261f1d5b-9bbf-437b-b110-ce09f23e400c" containerName="payload-processing"
Sep 11 17:52:32.961661 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.961651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="261f1d5b-9bbf-437b-b110-ce09f23e400c" containerName="payload-processing"
Sep 11 17:52:32.966076 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.966055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:32.970215 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.970195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-feb821e2-openshift-default-dockercfg-9mp77\""
Sep 11 17:52:32.978352 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:32.978325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"]
Sep 11 17:52:33.099394 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-data\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-token\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-socket\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-credential-socket\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-certs\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-podinfo\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phs65\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-kube-api-access-phs65\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099813 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b2deb-5aab-4667-a56a-35c45e506f58-istiod-ca-cert\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.099813 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.099691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-envoy\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200438 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-data\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-token\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-socket\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-credential-socket\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-certs\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-podinfo\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phs65\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-kube-api-access-phs65\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.200654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b2deb-5aab-4667-a56a-35c45e506f58-istiod-ca-cert\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.201033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-envoy\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.201033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-data\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.201033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.200979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-credential-socket\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.201326 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.201274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-certs\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.201447 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.201337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-socket\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.201541 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.201460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b2deb-5aab-4667-a56a-35c45e506f58-istiod-ca-cert\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.203081 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.203062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-envoy\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.203385 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.203367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-podinfo\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.209669 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.209643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-token\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.210576 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.210545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phs65\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-kube-api-access-phs65\") pod \"e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") " pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.278092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.277986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:33.567622 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.567595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"]
Sep 11 17:52:33.569962 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:52:33.569923    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6b2deb_5aab_4667_a56a_35c45e506f58.slice/crio-dcbcf0badabc95dc05586d483c92c69ee7155e746c466fb4d07cadd7b8aea4a2 WatchSource:0}: Error finding container dcbcf0badabc95dc05586d483c92c69ee7155e746c466fb4d07cadd7b8aea4a2: Status 404 returned error can't find the container with id dcbcf0badabc95dc05586d483c92c69ee7155e746c466fb4d07cadd7b8aea4a2
Sep 11 17:52:33.572737 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.572705    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:52:33.572820 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.572775    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:52:33.572859 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.572830    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:52:33.783240 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:33.783194    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261f1d5b-9bbf-437b-b110-ce09f23e400c" path="/var/lib/kubelet/pods/261f1d5b-9bbf-437b-b110-ce09f23e400c/volumes"
Sep 11 17:52:34.292894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:34.292851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf" event={"ID":"de6b2deb-5aab-4667-a56a-35c45e506f58","Type":"ContainerStarted","Data":"9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b"}
Sep 11 17:52:34.293406 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:34.292914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf" event={"ID":"de6b2deb-5aab-4667-a56a-35c45e506f58","Type":"ContainerStarted","Data":"dcbcf0badabc95dc05586d483c92c69ee7155e746c466fb4d07cadd7b8aea4a2"}
Sep 11 17:52:34.317053 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:34.317001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf" podStartSLOduration=2.316986782 podStartE2EDuration="2.316986782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:52:34.31554949 +0000 UTC m=+821.079004218" watchObservedRunningTime="2026-09-11 17:52:34.316986782 +0000 UTC m=+821.080441510"
Sep 11 17:52:35.278816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:35.278776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:35.284147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:35.284121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:35.298717 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:35.298688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:35.299617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:35.299592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:52:38.371150 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.371112    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"]
Sep 11 17:52:38.375618 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.375601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.378739 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.378713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-feb821e2\""
Sep 11 17:52:38.378847 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.378769    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-feb821e2-dockercfg-ksclk\""
Sep 11 17:52:38.387190 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.387160    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"]
Sep 11 17:52:38.456943 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.456886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvll\" (UniqueName: \"kubernetes.io/projected/9d8f4348-f568-41a6-b93a-1cdee5c0087a-kube-api-access-7pvll\") pod \"payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") " pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.457131 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.456994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8f4348-f568-41a6-b93a-1cdee5c0087a-plugins-config-volume\") pod \"payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") " pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.557848 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.557814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvll\" (UniqueName: \"kubernetes.io/projected/9d8f4348-f568-41a6-b93a-1cdee5c0087a-kube-api-access-7pvll\") pod \"payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") " pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.558059 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.557874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8f4348-f568-41a6-b93a-1cdee5c0087a-plugins-config-volume\") pod \"payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") " pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.558563 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.558544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8f4348-f568-41a6-b93a-1cdee5c0087a-plugins-config-volume\") pod \"payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") " pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.572190 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.572163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvll\" (UniqueName: \"kubernetes.io/projected/9d8f4348-f568-41a6-b93a-1cdee5c0087a-kube-api-access-7pvll\") pod \"payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") " pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.689245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.689155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:38.974709 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:38.974681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"]
Sep 11 17:52:38.975821 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:52:38.975794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8f4348_f568_41a6_b93a_1cdee5c0087a.slice/crio-89764e2ea4f03213826dee317e918c104281f7d48c882642280ba39fe1ab3ec4 WatchSource:0}: Error finding container 89764e2ea4f03213826dee317e918c104281f7d48c882642280ba39fe1ab3ec4: Status 404 returned error can't find the container with id 89764e2ea4f03213826dee317e918c104281f7d48c882642280ba39fe1ab3ec4
Sep 11 17:52:39.324891 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:39.324854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws" event={"ID":"9d8f4348-f568-41a6-b93a-1cdee5c0087a","Type":"ContainerStarted","Data":"73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a"}
Sep 11 17:52:39.325093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:39.324913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws" event={"ID":"9d8f4348-f568-41a6-b93a-1cdee5c0087a","Type":"ContainerStarted","Data":"89764e2ea4f03213826dee317e918c104281f7d48c882642280ba39fe1ab3ec4"}
Sep 11 17:52:39.325093 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:39.325066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:52:39.346057 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:39.346000    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws" podStartSLOduration=1.345984147 podStartE2EDuration="1.345984147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:52:39.34335191 +0000 UTC m=+826.106806638" watchObservedRunningTime="2026-09-11 17:52:39.345984147 +0000 UTC m=+826.109438873"
Sep 11 17:52:50.336143 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:52:50.336113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:53:07.833843 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:07.833805    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"]
Sep 11 17:53:07.834420 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:07.834062    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws" podUID="9d8f4348-f568-41a6-b93a-1cdee5c0087a" containerName="payload-processing" containerID="cri-o://73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a" gracePeriod=30
Sep 11 17:53:08.109427 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.109349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:53:08.266055 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.266013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pvll\" (UniqueName: \"kubernetes.io/projected/9d8f4348-f568-41a6-b93a-1cdee5c0087a-kube-api-access-7pvll\") pod \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") "
Sep 11 17:53:08.266243 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.266085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8f4348-f568-41a6-b93a-1cdee5c0087a-plugins-config-volume\") pod \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\" (UID: \"9d8f4348-f568-41a6-b93a-1cdee5c0087a\") "
Sep 11 17:53:08.266442 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.266414    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8f4348-f568-41a6-b93a-1cdee5c0087a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d8f4348-f568-41a6-b93a-1cdee5c0087a" (UID: "9d8f4348-f568-41a6-b93a-1cdee5c0087a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:53:08.268314 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.268278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8f4348-f568-41a6-b93a-1cdee5c0087a-kube-api-access-7pvll" (OuterVolumeSpecName: "kube-api-access-7pvll") pod "9d8f4348-f568-41a6-b93a-1cdee5c0087a" (UID: "9d8f4348-f568-41a6-b93a-1cdee5c0087a"). InnerVolumeSpecName "kube-api-access-7pvll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:08.367295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.367202    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvll\" (UniqueName: \"kubernetes.io/projected/9d8f4348-f568-41a6-b93a-1cdee5c0087a-kube-api-access-7pvll\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:08.367295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.367238    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d8f4348-f568-41a6-b93a-1cdee5c0087a-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:08.513781 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.513740    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d8f4348-f568-41a6-b93a-1cdee5c0087a" containerID="73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a" exitCode=0
Sep 11 17:53:08.513969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.513798    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"
Sep 11 17:53:08.513969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.513826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws" event={"ID":"9d8f4348-f568-41a6-b93a-1cdee5c0087a","Type":"ContainerDied","Data":"73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a"}
Sep 11 17:53:08.513969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.513879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws" event={"ID":"9d8f4348-f568-41a6-b93a-1cdee5c0087a","Type":"ContainerDied","Data":"89764e2ea4f03213826dee317e918c104281f7d48c882642280ba39fe1ab3ec4"}
Sep 11 17:53:08.513969 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.513904    2563 scope.go:117] "RemoveContainer" containerID="73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a"
Sep 11 17:53:08.525295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.525271    2563 scope.go:117] "RemoveContainer" containerID="73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a"
Sep 11 17:53:08.525749 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:53:08.525723    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a\": container with ID starting with 73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a not found: ID does not exist" containerID="73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a"
Sep 11 17:53:08.525864 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.525760    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a"} err="failed to get container status \"73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a\": rpc error: code = NotFound desc = could not find container \"73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a\": container with ID starting with 73cd57a75f6cde3d759e1e563e985c8adde5c3eee5ee9a08b26d3f77ed61907a not found: ID does not exist"
Sep 11 17:53:08.538942 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.538910    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"]
Sep 11 17:53:08.556510 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:08.556448    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-feb821e2-77c7fc868f-vkrws"]
Sep 11 17:53:09.227956 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.227920    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"]
Sep 11 17:53:09.228385 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.228157    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf" podUID="de6b2deb-5aab-4667-a56a-35c45e506f58" containerName="istio-proxy" containerID="cri-o://9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b" gracePeriod=30
Sep 11 17:53:09.388766 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.388738    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:53:09.481061 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.480977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b2deb-5aab-4667-a56a-35c45e506f58-istiod-ca-cert\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481061 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-socket\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481070    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-certs\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phs65\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-kube-api-access-phs65\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-data\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-credential-socket\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-token\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481270 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-envoy\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-podinfo\") pod \"de6b2deb-5aab-4667-a56a-35c45e506f58\" (UID: \"de6b2deb-5aab-4667-a56a-35c45e506f58\") "
Sep 11 17:53:09.481595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:09.481595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:09.481595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6b2deb-5aab-4667-a56a-35c45e506f58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:53:09.481595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-data" (OuterVolumeSpecName: "istio-data") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:09.481595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:09.482048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481749    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de6b2deb-5aab-4667-a56a-35c45e506f58-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.482048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481763    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.482048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481771    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.482048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481783    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.482048 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.481801    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.483597 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.483561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-kube-api-access-phs65" (OuterVolumeSpecName: "kube-api-access-phs65") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "kube-api-access-phs65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:09.483715 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.483604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-token" (OuterVolumeSpecName: "istio-token") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:09.483715 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.483620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:53:09.483792 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.483710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de6b2deb-5aab-4667-a56a-35c45e506f58" (UID: "de6b2deb-5aab-4667-a56a-35c45e506f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:53:09.522304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.522270    2563 generic.go:356] "Generic (PLEG): container finished" podID="de6b2deb-5aab-4667-a56a-35c45e506f58" containerID="9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b" exitCode=0
Sep 11 17:53:09.522440 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.522324    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"
Sep 11 17:53:09.522440 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.522360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf" event={"ID":"de6b2deb-5aab-4667-a56a-35c45e506f58","Type":"ContainerDied","Data":"9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b"}
Sep 11 17:53:09.522440 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.522412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf" event={"ID":"de6b2deb-5aab-4667-a56a-35c45e506f58","Type":"ContainerDied","Data":"dcbcf0badabc95dc05586d483c92c69ee7155e746c466fb4d07cadd7b8aea4a2"}
Sep 11 17:53:09.522440 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.522438    2563 scope.go:117] "RemoveContainer" containerID="9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b"
Sep 11 17:53:09.532533 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.532512    2563 scope.go:117] "RemoveContainer" containerID="9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b"
Sep 11 17:53:09.532822 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:53:09.532803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b\": container with ID starting with 9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b not found: ID does not exist" containerID="9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b"
Sep 11 17:53:09.532873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.532830    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b"} err="failed to get container status \"9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b\": rpc error: code = NotFound desc = could not find container \"9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b\": container with ID starting with 9fc8f0215df8de49f1c3d050d6268c68e741b70097265963682e64c6d0d4707b not found: ID does not exist"
Sep 11 17:53:09.548746 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.548715    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"]
Sep 11 17:53:09.552832 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.552806    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-feb821e2-openshift-default-7d46fd6c8-vgjcf"]
Sep 11 17:53:09.583331 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.583292    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.583331 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.583322    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.583331 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.583335    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de6b2deb-5aab-4667-a56a-35c45e506f58-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.583570 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.583347    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phs65\" (UniqueName: \"kubernetes.io/projected/de6b2deb-5aab-4667-a56a-35c45e506f58-kube-api-access-phs65\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:09.782982 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.782902    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8f4348-f568-41a6-b93a-1cdee5c0087a" path="/var/lib/kubelet/pods/9d8f4348-f568-41a6-b93a-1cdee5c0087a/volumes"
Sep 11 17:53:09.783360 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:09.783347    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de6b2deb-5aab-4667-a56a-35c45e506f58" path="/var/lib/kubelet/pods/de6b2deb-5aab-4667-a56a-35c45e506f58/volumes"
Sep 11 17:53:10.010837 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.010804    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"]
Sep 11 17:53:10.011276 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.011263    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8f4348-f568-41a6-b93a-1cdee5c0087a" containerName="payload-processing"
Sep 11 17:53:10.011320 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.011278    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8f4348-f568-41a6-b93a-1cdee5c0087a" containerName="payload-processing"
Sep 11 17:53:10.011320 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.011294    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de6b2deb-5aab-4667-a56a-35c45e506f58" containerName="istio-proxy"
Sep 11 17:53:10.011320 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.011299    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="de6b2deb-5aab-4667-a56a-35c45e506f58" containerName="istio-proxy"
Sep 11 17:53:10.011409 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.011366    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8f4348-f568-41a6-b93a-1cdee5c0087a" containerName="payload-processing"
Sep 11 17:53:10.011409 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.011377    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="de6b2deb-5aab-4667-a56a-35c45e506f58" containerName="istio-proxy"
Sep 11 17:53:10.014397 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.014371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.018244 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.018221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5f786284-openshift-default-dockercfg-srdg8\""
Sep 11 17:53:10.027628 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.027599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"]
Sep 11 17:53:10.088850 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.088813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-istio-token\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.088850 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.088847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b196d490-1a00-4d44-b639-3062ee522e21-istiod-ca-cert\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.088872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf6d\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-kube-api-access-7vf6d\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.088937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-envoy\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.089004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-credential-socket\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.089082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-certs\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089317 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.089152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-data\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089317 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.089183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b196d490-1a00-4d44-b639-3062ee522e21-istio-podinfo\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.089317 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.089266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-socket\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190556 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-certs\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-data\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b196d490-1a00-4d44-b639-3062ee522e21-istio-podinfo\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-socket\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-istio-token\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b196d490-1a00-4d44-b639-3062ee522e21-istiod-ca-cert\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.190817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf6d\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-kube-api-access-7vf6d\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-envoy\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-credential-socket\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-certs\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.190998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-data\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191188 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.191040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-socket\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191465 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.191427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-credential-socket\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.191585 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.191566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b196d490-1a00-4d44-b639-3062ee522e21-istiod-ca-cert\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.193236 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.193216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-envoy\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.193384 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.193270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b196d490-1a00-4d44-b639-3062ee522e21-istio-podinfo\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.204575 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.204551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-istio-token\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.208819 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.208794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf6d\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-kube-api-access-7vf6d\") pod \"e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") " pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.325279 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.325239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:10.617420 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.617384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"]
Sep 11 17:53:10.618311 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:53:10.618283    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb196d490_1a00_4d44_b639_3062ee522e21.slice/crio-511f20508f91dd09c44c67cf8411041bfd6df83bd56ef53aae90c57f63f84dc1 WatchSource:0}: Error finding container 511f20508f91dd09c44c67cf8411041bfd6df83bd56ef53aae90c57f63f84dc1: Status 404 returned error can't find the container with id 511f20508f91dd09c44c67cf8411041bfd6df83bd56ef53aae90c57f63f84dc1
Sep 11 17:53:10.620860 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.620826    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:53:10.620936 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.620894    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:53:10.620936 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:10.620927    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:53:11.542626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:11.542588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" event={"ID":"b196d490-1a00-4d44-b639-3062ee522e21","Type":"ContainerStarted","Data":"ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"}
Sep 11 17:53:11.543020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:11.542639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" event={"ID":"b196d490-1a00-4d44-b639-3062ee522e21","Type":"ContainerStarted","Data":"511f20508f91dd09c44c67cf8411041bfd6df83bd56ef53aae90c57f63f84dc1"}
Sep 11 17:53:11.572400 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:11.572345    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podStartSLOduration=2.572325461 podStartE2EDuration="2.572325461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:53:11.570021019 +0000 UTC m=+858.333475771" watchObservedRunningTime="2026-09-11 17:53:11.572325461 +0000 UTC m=+858.335780188"
Sep 11 17:53:12.326009 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:12.325976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:12.331338 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:12.331312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:12.548739 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:12.548709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:12.549595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:12.549576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:17.839216 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.839175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"]
Sep 11 17:53:17.842914 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.842888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:17.846365 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.846338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5f786284-serving-cert\""
Sep 11 17:53:17.859344 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.859314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"]
Sep 11 17:53:17.971057 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.971022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-maas-api-tls\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:17.971057 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.971061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-metrics-certs\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:17.971274 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.971158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ktc\" (UniqueName: \"kubernetes.io/projected/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-kube-api-access-p6ktc\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:17.971274 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:17.971194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-openshift-service-ca\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.072561 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.072524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ktc\" (UniqueName: \"kubernetes.io/projected/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-kube-api-access-p6ktc\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.072756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.072576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-openshift-service-ca\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.072756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.072669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-maas-api-tls\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.072756 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.072696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-metrics-certs\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.073246 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.073219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-openshift-service-ca\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.075293 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.075269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-maas-api-tls\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.075400 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.075306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-metrics-certs\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.085154 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.085122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ktc\" (UniqueName: \"kubernetes.io/projected/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-kube-api-access-p6ktc\") pod \"maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.154555 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.154427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:18.436644 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.436539    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"]
Sep 11 17:53:18.439427 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:53:18.439398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea3f3d2_36c8_4c77_b088_45c7bbd8ecf6.slice/crio-e3168e3c0f907471d255ea36a04cd6845ba32b8c35b4394df191533509d46377 WatchSource:0}: Error finding container e3168e3c0f907471d255ea36a04cd6845ba32b8c35b4394df191533509d46377: Status 404 returned error can't find the container with id e3168e3c0f907471d255ea36a04cd6845ba32b8c35b4394df191533509d46377
Sep 11 17:53:18.569380 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.569345    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"]
Sep 11 17:53:18.572332 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.572311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.575804 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.575785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5f786284-dockercfg-fm6rl\""
Sep 11 17:53:18.575804 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.575798    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5f786284\""
Sep 11 17:53:18.586461 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.586433    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"]
Sep 11 17:53:18.588079 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.588055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j" event={"ID":"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6","Type":"ContainerStarted","Data":"e3168e3c0f907471d255ea36a04cd6845ba32b8c35b4394df191533509d46377"}
Sep 11 17:53:18.679068 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.679030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2n7\" (UniqueName: \"kubernetes.io/projected/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-kube-api-access-jj2n7\") pod \"payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.679258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.679087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.779795 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.779703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2n7\" (UniqueName: \"kubernetes.io/projected/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-kube-api-access-jj2n7\") pod \"payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.779795 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.779743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.780559 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.780538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.789433 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.789399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2n7\" (UniqueName: \"kubernetes.io/projected/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-kube-api-access-jj2n7\") pod \"payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:18.884529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:18.884464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:19.162510 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.162450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"]
Sep 11 17:53:19.170263 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:53:19.170217    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ac3df0_38eb_4629_b1ba_20c9eaa5c391.slice/crio-f1ff24a1446cc222903a2d24b5eb3becab88d2646ed977853aaf56a3a8f50ce8 WatchSource:0}: Error finding container f1ff24a1446cc222903a2d24b5eb3becab88d2646ed977853aaf56a3a8f50ce8: Status 404 returned error can't find the container with id f1ff24a1446cc222903a2d24b5eb3becab88d2646ed977853aaf56a3a8f50ce8
Sep 11 17:53:19.595669 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.595620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j" event={"ID":"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6","Type":"ContainerStarted","Data":"b5cd3c72fcd53cab63430b2bcd5ee7acbbb4ad2609a804ac565b00841e153c4d"}
Sep 11 17:53:19.595889 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.595695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:19.597410 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.597380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8" event={"ID":"72ac3df0-38eb-4629-b1ba-20c9eaa5c391","Type":"ContainerStarted","Data":"797bb7dc2f0facdf40ff44634ae94c7b6c0cd6d56d7c9858ae66313a9fefe0d8"}
Sep 11 17:53:19.597569 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.597432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:19.597569 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.597449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8" event={"ID":"72ac3df0-38eb-4629-b1ba-20c9eaa5c391","Type":"ContainerStarted","Data":"f1ff24a1446cc222903a2d24b5eb3becab88d2646ed977853aaf56a3a8f50ce8"}
Sep 11 17:53:19.618168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.618116    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j" podStartSLOduration=2.244223905 podStartE2EDuration="2.618097751s" totalImagesPullingTime="373.873846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:53:19.617022032 +0000 UTC m=+866.380476760" watchObservedRunningTime="2026-09-11 17:53:19.618097751 +0000 UTC m=+866.381552479"
Sep 11 17:53:19.638483 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:19.638424    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8" podStartSLOduration=1.63840634 podStartE2EDuration="1.63840634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:53:19.635861031 +0000 UTC m=+866.399315758" watchObservedRunningTime="2026-09-11 17:53:19.63840634 +0000 UTC m=+866.401861066"
Sep 11 17:53:25.610278 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:25.610250    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:30.607038 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:30.606998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:36.704203 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:36.704168    2563 generic.go:356] "Generic (PLEG): container finished" podID="b196d490-1a00-4d44-b639-3062ee522e21" containerID="ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c" exitCode=0
Sep 11 17:53:36.704638 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:36.704237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" event={"ID":"b196d490-1a00-4d44-b639-3062ee522e21","Type":"ContainerDied","Data":"ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"}
Sep 11 17:53:36.704729 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:36.704714    2563 scope.go:117] "RemoveContainer" containerID="ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"
Sep 11 17:53:36.705863 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:36.705838    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:53:36.705938 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:36.705904    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:53:36.705975 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:36.705934    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:53:37.712662 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:37.712624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" event={"ID":"b196d490-1a00-4d44-b639-3062ee522e21","Type":"ContainerStarted","Data":"e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a"}
Sep 11 17:53:38.326341 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:38.326305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:38.327412 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:38.327385    2563 patch_prober.go:28] interesting pod/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 17:53:38.327545 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:38.327445    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 17:53:39.325723 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:39.325685    2563 patch_prober.go:28] interesting pod/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 17:53:39.326108 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:39.325738    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 17:53:40.325956 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:40.325921    2563 patch_prober.go:28] interesting pod/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 17:53:40.326350 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:40.325979    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 17:53:41.326655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:41.326612    2563 patch_prober.go:28] interesting pod/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 17:53:41.327046 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:41.326680    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 17:53:42.326676 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:42.326636    2563 patch_prober.go:28] interesting pod/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 17:53:42.327142 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:42.326708    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 17:53:42.549043 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:42.549005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:43.330435 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:43.330404    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:43.331668 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:43.331649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:53:53.750143 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:53.750059    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:53:53.751880 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:53.751852    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:53:54.623011 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.622976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"]
Sep 11 17:53:54.623278 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.623217    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j" podUID="7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" containerName="maas-api" containerID="cri-o://b5cd3c72fcd53cab63430b2bcd5ee7acbbb4ad2609a804ac565b00841e153c4d" gracePeriod=30
Sep 11 17:53:54.693866 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.693828    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"]
Sep 11 17:53:54.694113 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.694084    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8" podUID="72ac3df0-38eb-4629-b1ba-20c9eaa5c391" containerName="payload-pre-processing" containerID="cri-o://797bb7dc2f0facdf40ff44634ae94c7b6c0cd6d56d7c9858ae66313a9fefe0d8" gracePeriod=30
Sep 11 17:53:54.825716 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.825654    2563 generic.go:356] "Generic (PLEG): container finished" podID="72ac3df0-38eb-4629-b1ba-20c9eaa5c391" containerID="797bb7dc2f0facdf40ff44634ae94c7b6c0cd6d56d7c9858ae66313a9fefe0d8" exitCode=0
Sep 11 17:53:54.827854 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.827709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8" event={"ID":"72ac3df0-38eb-4629-b1ba-20c9eaa5c391","Type":"ContainerDied","Data":"797bb7dc2f0facdf40ff44634ae94c7b6c0cd6d56d7c9858ae66313a9fefe0d8"}
Sep 11 17:53:54.828832 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.828805    2563 generic.go:356] "Generic (PLEG): container finished" podID="7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" containerID="b5cd3c72fcd53cab63430b2bcd5ee7acbbb4ad2609a804ac565b00841e153c4d" exitCode=0
Sep 11 17:53:54.828975 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.828900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j" event={"ID":"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6","Type":"ContainerDied","Data":"b5cd3c72fcd53cab63430b2bcd5ee7acbbb4ad2609a804ac565b00841e153c4d"}
Sep 11 17:53:54.874094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.873958    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"]
Sep 11 17:53:54.875006 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:54.874345    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl" podUID="72437013-8ec6-4dcb-88f4-06f601ea8608" containerName="payload-pre-processing" containerID="cri-o://b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1" gracePeriod=30
Sep 11 17:53:55.014997 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.014968    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:55.018644 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.018621    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.081456    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-plugins-config-volume\") pod \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") "
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.081523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-metrics-certs\") pod \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") "
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.081608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj2n7\" (UniqueName: \"kubernetes.io/projected/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-kube-api-access-jj2n7\") pod \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\" (UID: \"72ac3df0-38eb-4629-b1ba-20c9eaa5c391\") "
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.081656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-maas-api-tls\") pod \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") "
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.081696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-openshift-service-ca\") pod \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") "
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.081741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6ktc\" (UniqueName: \"kubernetes.io/projected/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-kube-api-access-p6ktc\") pod \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\" (UID: \"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6\") "
Sep 11 17:53:55.082122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.082030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72ac3df0-38eb-4629-b1ba-20c9eaa5c391" (UID: "72ac3df0-38eb-4629-b1ba-20c9eaa5c391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:53:55.082643 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.082367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" (UID: "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:53:55.087467 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.087409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" (UID: "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:53:55.089117 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.089088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-kube-api-access-jj2n7" (OuterVolumeSpecName: "kube-api-access-jj2n7") pod "72ac3df0-38eb-4629-b1ba-20c9eaa5c391" (UID: "72ac3df0-38eb-4629-b1ba-20c9eaa5c391"). InnerVolumeSpecName "kube-api-access-jj2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:55.089250 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.089195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" (UID: "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:53:55.089433 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.089401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-kube-api-access-p6ktc" (OuterVolumeSpecName: "kube-api-access-p6ktc") pod "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" (UID: "7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6"). InnerVolumeSpecName "kube-api-access-p6ktc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:55.101061 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.100984    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"]
Sep 11 17:53:55.101776 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.101437    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" containerID="cri-o://e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a" gracePeriod=30
Sep 11 17:53:55.166595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.166568    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:53:55.183302 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72437013-8ec6-4dcb-88f4-06f601ea8608-plugins-config-volume\") pod \"72437013-8ec6-4dcb-88f4-06f601ea8608\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") "
Sep 11 17:53:55.183502 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183326    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7npf\" (UniqueName: \"kubernetes.io/projected/72437013-8ec6-4dcb-88f4-06f601ea8608-kube-api-access-b7npf\") pod \"72437013-8ec6-4dcb-88f4-06f601ea8608\" (UID: \"72437013-8ec6-4dcb-88f4-06f601ea8608\") "
Sep 11 17:53:55.183726 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72437013-8ec6-4dcb-88f4-06f601ea8608-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72437013-8ec6-4dcb-88f4-06f601ea8608" (UID: "72437013-8ec6-4dcb-88f4-06f601ea8608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:53:55.183726 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183719    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2n7\" (UniqueName: \"kubernetes.io/projected/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-kube-api-access-jj2n7\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.183921 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183742    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.183921 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183756    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.183921 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183769    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ktc\" (UniqueName: \"kubernetes.io/projected/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-kube-api-access-p6ktc\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.183921 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183782    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ac3df0-38eb-4629-b1ba-20c9eaa5c391-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.183921 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.183794    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.186043 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.186013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72437013-8ec6-4dcb-88f4-06f601ea8608-kube-api-access-b7npf" (OuterVolumeSpecName: "kube-api-access-b7npf") pod "72437013-8ec6-4dcb-88f4-06f601ea8608" (UID: "72437013-8ec6-4dcb-88f4-06f601ea8608"). InnerVolumeSpecName "kube-api-access-b7npf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:53:55.288513 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.288358    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72437013-8ec6-4dcb-88f4-06f601ea8608-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.288513 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.288400    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7npf\" (UniqueName: \"kubernetes.io/projected/72437013-8ec6-4dcb-88f4-06f601ea8608-kube-api-access-b7npf\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:53:55.837058 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.837025    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"
Sep 11 17:53:55.837543 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.837058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j" event={"ID":"7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6","Type":"ContainerDied","Data":"e3168e3c0f907471d255ea36a04cd6845ba32b8c35b4394df191533509d46377"}
Sep 11 17:53:55.837543 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.837136    2563 scope.go:117] "RemoveContainer" containerID="b5cd3c72fcd53cab63430b2bcd5ee7acbbb4ad2609a804ac565b00841e153c4d"
Sep 11 17:53:55.838650 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.838624    2563 generic.go:356] "Generic (PLEG): container finished" podID="72437013-8ec6-4dcb-88f4-06f601ea8608" containerID="b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1" exitCode=0
Sep 11 17:53:55.838771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.838669    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"
Sep 11 17:53:55.838771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.838692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl" event={"ID":"72437013-8ec6-4dcb-88f4-06f601ea8608","Type":"ContainerDied","Data":"b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1"}
Sep 11 17:53:55.838771 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.838737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl" event={"ID":"72437013-8ec6-4dcb-88f4-06f601ea8608","Type":"ContainerDied","Data":"e16f554c6982a2979b98cd8b613c817eabe67fc7a6dbeed4665f3c6c0dc42638"}
Sep 11 17:53:55.840577 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.840553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8" event={"ID":"72ac3df0-38eb-4629-b1ba-20c9eaa5c391","Type":"ContainerDied","Data":"f1ff24a1446cc222903a2d24b5eb3becab88d2646ed977853aaf56a3a8f50ce8"}
Sep 11 17:53:55.840677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.840594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"
Sep 11 17:53:55.848844 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.848827    2563 scope.go:117] "RemoveContainer" containerID="b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1"
Sep 11 17:53:55.858414 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.858395    2563 scope.go:117] "RemoveContainer" containerID="b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1"
Sep 11 17:53:55.858811 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:53:55.858793    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1\": container with ID starting with b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1 not found: ID does not exist" containerID="b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1"
Sep 11 17:53:55.858882 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.858822    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1"} err="failed to get container status \"b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1\": rpc error: code = NotFound desc = could not find container \"b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1\": container with ID starting with b153648a76a06ec23ef26273054a62a68895a8088f0c324d46302a0bcc1bc5e1 not found: ID does not exist"
Sep 11 17:53:55.858882 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.858843    2563 scope.go:117] "RemoveContainer" containerID="797bb7dc2f0facdf40ff44634ae94c7b6c0cd6d56d7c9858ae66313a9fefe0d8"
Sep 11 17:53:55.889444 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.889402    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"]
Sep 11 17:53:55.894681 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.894647    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-37899e-6c4cd874f4-jpwwl"]
Sep 11 17:53:55.913089 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.913059    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"]
Sep 11 17:53:55.917011 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.916982    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5f786284-84dbd8cbc4-xtbz8"]
Sep 11 17:53:55.942842 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.942809    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"]
Sep 11 17:53:55.949967 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:55.949937    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5f786284-55bcb984c7-vlm5j"]
Sep 11 17:53:57.782070 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:57.782031    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72437013-8ec6-4dcb-88f4-06f601ea8608" path="/var/lib/kubelet/pods/72437013-8ec6-4dcb-88f4-06f601ea8608/volumes"
Sep 11 17:53:57.782654 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:57.782398    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ac3df0-38eb-4629-b1ba-20c9eaa5c391" path="/var/lib/kubelet/pods/72ac3df0-38eb-4629-b1ba-20c9eaa5c391/volumes"
Sep 11 17:53:57.782794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:57.782779    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" path="/var/lib/kubelet/pods/7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6/volumes"
Sep 11 17:53:58.333817 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:58.333779    2563 patch_prober.go:28] interesting pod/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 17:53:58.334005 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:53:58.333857    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:54:00.359685 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.359653    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:54:00.440295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vf6d\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-kube-api-access-7vf6d\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-certs\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440295 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-socket\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b196d490-1a00-4d44-b639-3062ee522e21-istio-podinfo\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-credential-socket\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-envoy\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440591 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b196d490-1a00-4d44-b639-3062ee522e21-istiod-ca-cert\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440798 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:00.440798 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:00.440798 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440682    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:00.440798 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-istio-token\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.440798 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.440736    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-data\") pod \"b196d490-1a00-4d44-b639-3062ee522e21\" (UID: \"b196d490-1a00-4d44-b639-3062ee522e21\") "
Sep 11 17:54:00.441033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-data" (OuterVolumeSpecName: "istio-data") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:00.441033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b196d490-1a00-4d44-b639-3062ee522e21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:54:00.441191 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441115    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.441191 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441135    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.441191 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441150    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.441191 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441164    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.441191 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.441178    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b196d490-1a00-4d44-b639-3062ee522e21-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.442794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.442770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b196d490-1a00-4d44-b639-3062ee522e21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:54:00.442899 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.442818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:54:00.443148 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.443130    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-kube-api-access-7vf6d" (OuterVolumeSpecName: "kube-api-access-7vf6d") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "kube-api-access-7vf6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:00.443213 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.443132    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-istio-token" (OuterVolumeSpecName: "istio-token") pod "b196d490-1a00-4d44-b639-3062ee522e21" (UID: "b196d490-1a00-4d44-b639-3062ee522e21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:54:00.542053 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.542014    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.542249 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.542062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vf6d\" (UniqueName: \"kubernetes.io/projected/b196d490-1a00-4d44-b639-3062ee522e21-kube-api-access-7vf6d\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.542249 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.542078    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b196d490-1a00-4d44-b639-3062ee522e21-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.542249 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.542090    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b196d490-1a00-4d44-b639-3062ee522e21-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:54:00.887385 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.887349    2563 generic.go:356] "Generic (PLEG): container finished" podID="b196d490-1a00-4d44-b639-3062ee522e21" containerID="e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a" exitCode=0
Sep 11 17:54:00.887626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.887421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" event={"ID":"b196d490-1a00-4d44-b639-3062ee522e21","Type":"ContainerDied","Data":"e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a"}
Sep 11 17:54:00.887626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.887478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq" event={"ID":"b196d490-1a00-4d44-b639-3062ee522e21","Type":"ContainerDied","Data":"511f20508f91dd09c44c67cf8411041bfd6df83bd56ef53aae90c57f63f84dc1"}
Sep 11 17:54:00.887626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.887428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"
Sep 11 17:54:00.887626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.887529    2563 scope.go:117] "RemoveContainer" containerID="e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a"
Sep 11 17:54:00.898677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.898659    2563 scope.go:117] "RemoveContainer" containerID="ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"
Sep 11 17:54:00.907672 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.907652    2563 scope.go:117] "RemoveContainer" containerID="e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a"
Sep 11 17:54:00.907966 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:54:00.907941    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a\": container with ID starting with e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a not found: ID does not exist" containerID="e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a"
Sep 11 17:54:00.908033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.907976    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a"} err="failed to get container status \"e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a\": rpc error: code = NotFound desc = could not find container \"e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a\": container with ID starting with e42c2001e4f2218b1edfd2d9f4f14ae1cc459a888771979b171d1c0f5f0f9b6a not found: ID does not exist"
Sep 11 17:54:00.908033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.907994    2563 scope.go:117] "RemoveContainer" containerID="ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"
Sep 11 17:54:00.908304 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:54:00.908277    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c\": container with ID starting with ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c not found: ID does not exist" containerID="ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"
Sep 11 17:54:00.908405 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.908313    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c"} err="failed to get container status \"ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c\": rpc error: code = NotFound desc = could not find container \"ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c\": container with ID starting with ac1eb141985e021936bf3f354af893b7ec351851c85f19aa7b132cb28f3dc51c not found: ID does not exist"
Sep 11 17:54:00.912750 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.912727    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"]
Sep 11 17:54:00.917695 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:00.917674    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5f786284-openshift-default-7c9858fb96-pkpmq"]
Sep 11 17:54:01.782172 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:01.782126    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b196d490-1a00-4d44-b639-3062ee522e21" path="/var/lib/kubelet/pods/b196d490-1a00-4d44-b639-3062ee522e21/volumes"
Sep 11 17:54:46.769732 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.769687    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"]
Sep 11 17:54:46.770879 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770850    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" containerName="maas-api"
Sep 11 17:54:46.770879 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770879    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" containerName="maas-api"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770898    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72437013-8ec6-4dcb-88f4-06f601ea8608" containerName="payload-pre-processing"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770908    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72437013-8ec6-4dcb-88f4-06f601ea8608" containerName="payload-pre-processing"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770917    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ac3df0-38eb-4629-b1ba-20c9eaa5c391" containerName="payload-pre-processing"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770944    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ac3df0-38eb-4629-b1ba-20c9eaa5c391" containerName="payload-pre-processing"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770953    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.770961    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.771062    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea3f3d2-36c8-4c77-b088-45c7bbd8ecf6" containerName="maas-api"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.771075    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ac3df0-38eb-4629-b1ba-20c9eaa5c391" containerName="payload-pre-processing"
Sep 11 17:54:46.771086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.771085    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy"
Sep 11 17:54:46.771645 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.771094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72437013-8ec6-4dcb-88f4-06f601ea8608" containerName="payload-pre-processing"
Sep 11 17:54:46.771645 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.771102    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b196d490-1a00-4d44-b639-3062ee522e21" containerName="istio-proxy"
Sep 11 17:54:46.774789 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.774761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:46.780789 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.780755    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-104872-dockercfg-jvsfr\""
Sep 11 17:54:46.781094 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.781060    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-104872\""
Sep 11 17:54:46.787407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.786995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"]
Sep 11 17:54:46.927060 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.927021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2g2\" (UniqueName: \"kubernetes.io/projected/b53598b2-74b9-4d44-b134-ebf177756004-kube-api-access-kn2g2\") pod \"payload-processing-e2e-explicit-104872-8546c44b65-mbr27\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") " pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:46.927304 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:46.927281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53598b2-74b9-4d44-b134-ebf177756004-plugins-config-volume\") pod \"payload-processing-e2e-explicit-104872-8546c44b65-mbr27\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") " pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:47.028506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:47.028391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53598b2-74b9-4d44-b134-ebf177756004-plugins-config-volume\") pod \"payload-processing-e2e-explicit-104872-8546c44b65-mbr27\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") " pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:47.028506 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:47.028480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2g2\" (UniqueName: \"kubernetes.io/projected/b53598b2-74b9-4d44-b134-ebf177756004-kube-api-access-kn2g2\") pod \"payload-processing-e2e-explicit-104872-8546c44b65-mbr27\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") " pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:47.029132 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:47.029111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53598b2-74b9-4d44-b134-ebf177756004-plugins-config-volume\") pod \"payload-processing-e2e-explicit-104872-8546c44b65-mbr27\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") " pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:47.039138 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:47.039111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2g2\" (UniqueName: \"kubernetes.io/projected/b53598b2-74b9-4d44-b134-ebf177756004-kube-api-access-kn2g2\") pod \"payload-processing-e2e-explicit-104872-8546c44b65-mbr27\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") " pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:47.094339 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:47.094295    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:47.378328 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:47.378301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"]
Sep 11 17:54:47.380272 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:54:47.380243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53598b2_74b9_4d44_b134_ebf177756004.slice/crio-764eae297b2a72c2c9a85b92cabc73a52cdde4877554d10cf7ca18e6c5d34cf1 WatchSource:0}: Error finding container 764eae297b2a72c2c9a85b92cabc73a52cdde4877554d10cf7ca18e6c5d34cf1: Status 404 returned error can't find the container with id 764eae297b2a72c2c9a85b92cabc73a52cdde4877554d10cf7ca18e6c5d34cf1
Sep 11 17:54:48.172327 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:48.172293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27" event={"ID":"b53598b2-74b9-4d44-b134-ebf177756004","Type":"ContainerStarted","Data":"b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009"}
Sep 11 17:54:48.172745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:48.172361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:54:48.172745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:48.172386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27" event={"ID":"b53598b2-74b9-4d44-b134-ebf177756004","Type":"ContainerStarted","Data":"764eae297b2a72c2c9a85b92cabc73a52cdde4877554d10cf7ca18e6c5d34cf1"}
Sep 11 17:54:48.191962 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:48.191910    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27" podStartSLOduration=2.191895508 podStartE2EDuration="2.191895508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:48.189289318 +0000 UTC m=+954.952744046" watchObservedRunningTime="2026-09-11 17:54:48.191895508 +0000 UTC m=+954.955350282"
Sep 11 17:54:59.183011 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:54:59.182979    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:55:32.909717 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:32.909686    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"]
Sep 11 17:55:32.913607 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:32.913580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:32.917192 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:32.917172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fcdaf27d-openshift-default-dockercfg-crvnr\""
Sep 11 17:55:32.930056 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:32.930022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"]
Sep 11 17:55:33.005461 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a231928-c913-4146-84dd-955e6a4ab759-istiod-ca-cert\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005686 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-socket\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005686 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-certs\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005686 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a231928-c913-4146-84dd-955e6a4ab759-istio-podinfo\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005686 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-istio-token\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005926 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c669\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-kube-api-access-6c669\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005926 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-credential-socket\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005926 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-data\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.005926 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.005856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-envoy\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.106786 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c669\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-kube-api-access-6c669\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.106968 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-credential-socket\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.106968 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-data\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.106968 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-envoy\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.106968 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a231928-c913-4146-84dd-955e6a4ab759-istiod-ca-cert\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.106968 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-socket\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.106994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-certs\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a231928-c913-4146-84dd-955e6a4ab759-istio-podinfo\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-istio-token\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107445 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-socket\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107547 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-credential-socket\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107617 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-data\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107684 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-certs\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.107827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.107804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a231928-c913-4146-84dd-955e6a4ab759-istiod-ca-cert\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.109607 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.109581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-envoy\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.109806 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.109791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a231928-c913-4146-84dd-955e6a4ab759-istio-podinfo\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.118679 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.118648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c669\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-kube-api-access-6c669\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.118799 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.118781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-istio-token\") pod \"e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") " pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.224036 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.223924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:33.514746 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.514713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"]
Sep 11 17:55:33.516591 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:55:33.516561    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a231928_c913_4146_84dd_955e6a4ab759.slice/crio-49bb0a2fa153fc53dc4832af04b2bc2218f0b867cbd71d207dcf6f1395ae3db8 WatchSource:0}: Error finding container 49bb0a2fa153fc53dc4832af04b2bc2218f0b867cbd71d207dcf6f1395ae3db8: Status 404 returned error can't find the container with id 49bb0a2fa153fc53dc4832af04b2bc2218f0b867cbd71d207dcf6f1395ae3db8
Sep 11 17:55:33.518846 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.518812    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:55:33.518931 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.518879    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:55:33.518931 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:33.518908    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:55:34.448091 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:34.448052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" event={"ID":"4a231928-c913-4146-84dd-955e6a4ab759","Type":"ContainerStarted","Data":"51321b0923c7478e1c988bc5ed19d0a8a148f32cfd48f4ab04510a2f568f596f"}
Sep 11 17:55:34.448469 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:34.448114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" event={"ID":"4a231928-c913-4146-84dd-955e6a4ab759","Type":"ContainerStarted","Data":"49bb0a2fa153fc53dc4832af04b2bc2218f0b867cbd71d207dcf6f1395ae3db8"}
Sep 11 17:55:34.480131 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:34.480069    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podStartSLOduration=2.480049629 podStartE2EDuration="2.480049629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:34.472648449 +0000 UTC m=+1001.236103176" watchObservedRunningTime="2026-09-11 17:55:34.480049629 +0000 UTC m=+1001.243504356"
Sep 11 17:55:35.224450 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:35.224399    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:35.229763 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:35.229734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:35.454110 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:35.454080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:35.455017 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:35.454999    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:55:43.774992 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.774956    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"]
Sep 11 17:55:43.779573 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.779549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.782711 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.782686    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fcdaf27d-serving-cert\""
Sep 11 17:55:43.797257 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.797223    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"]
Sep 11 17:55:43.814326 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.814295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-metrics-certs\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.814531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.814356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c91276-1dcf-45e2-8a20-a0183d96f5d3-openshift-service-ca\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.814531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.814413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcvc\" (UniqueName: \"kubernetes.io/projected/35c91276-1dcf-45e2-8a20-a0183d96f5d3-kube-api-access-fzcvc\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.814531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.814455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-maas-api-tls\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.915621 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.915586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-metrics-certs\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.915818 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.915655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c91276-1dcf-45e2-8a20-a0183d96f5d3-openshift-service-ca\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.915818 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.915694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcvc\" (UniqueName: \"kubernetes.io/projected/35c91276-1dcf-45e2-8a20-a0183d96f5d3-kube-api-access-fzcvc\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.915818 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.915751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-maas-api-tls\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.916402 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.916376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c91276-1dcf-45e2-8a20-a0183d96f5d3-openshift-service-ca\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.918193 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.918170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-maas-api-tls\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.918306 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.918207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-metrics-certs\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:43.924595 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:43.924569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcvc\" (UniqueName: \"kubernetes.io/projected/35c91276-1dcf-45e2-8a20-a0183d96f5d3-kube-api-access-fzcvc\") pod \"maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:44.091213 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:44.091175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:44.377050 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:44.377018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"]
Sep 11 17:55:44.378011 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:55:44.377984    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c91276_1dcf_45e2_8a20_a0183d96f5d3.slice/crio-ede683a8911bdfac58a0b81c72a27c6880511c24898a0a5f65300f239ac82ea8 WatchSource:0}: Error finding container ede683a8911bdfac58a0b81c72a27c6880511c24898a0a5f65300f239ac82ea8: Status 404 returned error can't find the container with id ede683a8911bdfac58a0b81c72a27c6880511c24898a0a5f65300f239ac82ea8
Sep 11 17:55:44.508068 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:44.508024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5" event={"ID":"35c91276-1dcf-45e2-8a20-a0183d96f5d3","Type":"ContainerStarted","Data":"ede683a8911bdfac58a0b81c72a27c6880511c24898a0a5f65300f239ac82ea8"}
Sep 11 17:55:45.516248 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:45.516212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5" event={"ID":"35c91276-1dcf-45e2-8a20-a0183d96f5d3","Type":"ContainerStarted","Data":"5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37"}
Sep 11 17:55:45.516677 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:45.516301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:55:45.540578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:45.540522    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5" podStartSLOduration=2.075937097 podStartE2EDuration="2.540505536s" totalImagesPullingTime="464.568439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:45.539004816 +0000 UTC m=+1012.302459546" watchObservedRunningTime="2026-09-11 17:55:45.540505536 +0000 UTC m=+1012.303960253"
Sep 11 17:55:51.527894 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:55:51.527863    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:56:05.657770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:05.656588    2563 patch_prober.go:28] interesting pod/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:50394->10.134.0.59:15021: read: connection reset by peer" start-of-body=
Sep 11 17:56:05.657770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:05.656669    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:50394->10.134.0.59:15021: read: connection reset by peer"
Sep 11 17:56:06.666122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:06.666085    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a231928-c913-4146-84dd-955e6a4ab759" containerID="51321b0923c7478e1c988bc5ed19d0a8a148f32cfd48f4ab04510a2f568f596f" exitCode=0
Sep 11 17:56:06.666541 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:06.666135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" event={"ID":"4a231928-c913-4146-84dd-955e6a4ab759","Type":"ContainerDied","Data":"51321b0923c7478e1c988bc5ed19d0a8a148f32cfd48f4ab04510a2f568f596f"}
Sep 11 17:56:06.666626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:06.666610    2563 scope.go:117] "RemoveContainer" containerID="51321b0923c7478e1c988bc5ed19d0a8a148f32cfd48f4ab04510a2f568f596f"
Sep 11 17:56:06.667875 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:06.667835    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:06.667964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:06.667936    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:06.668020 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:06.667978    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:07.678983 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:07.678941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" event={"ID":"4a231928-c913-4146-84dd-955e6a4ab759","Type":"ContainerStarted","Data":"1d68d4d70c52e1eb835976cd711066f937d35de531010f04b70f856cf7b4beb7"}
Sep 11 17:56:08.224310 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:08.224263    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:56:08.224568 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:08.224472    2563 patch_prober.go:28] interesting pod/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:56:08.224651 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:08.224566    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 17:56:09.224669 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:09.224632    2563 patch_prober.go:28] interesting pod/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:56:09.225061 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:09.224708    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 17:56:10.224578 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:10.224542    2563 patch_prober.go:28] interesting pod/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:56:10.224758 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:10.224596    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 17:56:11.224637 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:11.224604    2563 patch_prober.go:28] interesting pod/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:56:11.224816 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:11.224659    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 17:56:12.224435 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:12.224394    2563 patch_prober.go:28] interesting pod/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 11 17:56:12.224655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:12.224460    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 11 17:56:13.228374 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:13.228336    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:56:13.228877 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:13.228682    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:56:13.229341 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:13.229324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:56:28.225254 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.225212    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"]
Sep 11 17:56:28.225852 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.225465    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5" podUID="35c91276-1dcf-45e2-8a20-a0183d96f5d3" containerName="maas-api" containerID="cri-o://5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37" gracePeriod=30
Sep 11 17:56:28.497173 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.496718    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:56:28.590179 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.590141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzcvc\" (UniqueName: \"kubernetes.io/projected/35c91276-1dcf-45e2-8a20-a0183d96f5d3-kube-api-access-fzcvc\") pod \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") "
Sep 11 17:56:28.590359 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.590255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-maas-api-tls\") pod \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") "
Sep 11 17:56:28.590359 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.590284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c91276-1dcf-45e2-8a20-a0183d96f5d3-openshift-service-ca\") pod \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") "
Sep 11 17:56:28.590441 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.590384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-metrics-certs\") pod \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\" (UID: \"35c91276-1dcf-45e2-8a20-a0183d96f5d3\") "
Sep 11 17:56:28.590804 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.590768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c91276-1dcf-45e2-8a20-a0183d96f5d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35c91276-1dcf-45e2-8a20-a0183d96f5d3" (UID: "35c91276-1dcf-45e2-8a20-a0183d96f5d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:28.592731 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.592696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c91276-1dcf-45e2-8a20-a0183d96f5d3-kube-api-access-fzcvc" (OuterVolumeSpecName: "kube-api-access-fzcvc") pod "35c91276-1dcf-45e2-8a20-a0183d96f5d3" (UID: "35c91276-1dcf-45e2-8a20-a0183d96f5d3"). InnerVolumeSpecName "kube-api-access-fzcvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:28.592731 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.592709    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "35c91276-1dcf-45e2-8a20-a0183d96f5d3" (UID: "35c91276-1dcf-45e2-8a20-a0183d96f5d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:28.592878 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.592770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35c91276-1dcf-45e2-8a20-a0183d96f5d3" (UID: "35c91276-1dcf-45e2-8a20-a0183d96f5d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:28.691965 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.691925    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:28.691965 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.691956    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcvc\" (UniqueName: \"kubernetes.io/projected/35c91276-1dcf-45e2-8a20-a0183d96f5d3-kube-api-access-fzcvc\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:28.691965 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.691965    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35c91276-1dcf-45e2-8a20-a0183d96f5d3-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:28.691965 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.691974    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35c91276-1dcf-45e2-8a20-a0183d96f5d3-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:28.822908 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.822871    2563 generic.go:356] "Generic (PLEG): container finished" podID="35c91276-1dcf-45e2-8a20-a0183d96f5d3" containerID="5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37" exitCode=0
Sep 11 17:56:28.823053 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.822922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"
Sep 11 17:56:28.823053 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.822957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5" event={"ID":"35c91276-1dcf-45e2-8a20-a0183d96f5d3","Type":"ContainerDied","Data":"5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37"}
Sep 11 17:56:28.823053 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.823014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5" event={"ID":"35c91276-1dcf-45e2-8a20-a0183d96f5d3","Type":"ContainerDied","Data":"ede683a8911bdfac58a0b81c72a27c6880511c24898a0a5f65300f239ac82ea8"}
Sep 11 17:56:28.823053 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.823041    2563 scope.go:117] "RemoveContainer" containerID="5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37"
Sep 11 17:56:28.833632 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.833613    2563 scope.go:117] "RemoveContainer" containerID="5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37"
Sep 11 17:56:28.833935 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:56:28.833915    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37\": container with ID starting with 5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37 not found: ID does not exist" containerID="5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37"
Sep 11 17:56:28.834004 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.833945    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37"} err="failed to get container status \"5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37\": rpc error: code = NotFound desc = could not find container \"5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37\": container with ID starting with 5927393eb8b124f475714af266cfd27ef9f88599e14c096cefb2b8e6b23cab37 not found: ID does not exist"
Sep 11 17:56:28.859069 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.859037    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"]
Sep 11 17:56:28.884206 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:28.884168    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fcdaf27d-6dbccbcfcd-nvcd5"]
Sep 11 17:56:29.783446 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:29.783405    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35c91276-1dcf-45e2-8a20-a0183d96f5d3" path="/var/lib/kubelet/pods/35c91276-1dcf-45e2-8a20-a0183d96f5d3/volumes"
Sep 11 17:56:35.692531 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:35.692482    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"]
Sep 11 17:56:35.693031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:35.692740    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy" containerID="cri-o://1d68d4d70c52e1eb835976cd711066f937d35de531010f04b70f856cf7b4beb7" gracePeriod=30
Sep 11 17:56:40.898087 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:40.898044    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a231928-c913-4146-84dd-955e6a4ab759" containerID="1d68d4d70c52e1eb835976cd711066f937d35de531010f04b70f856cf7b4beb7" exitCode=0
Sep 11 17:56:40.898579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:40.898098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" event={"ID":"4a231928-c913-4146-84dd-955e6a4ab759","Type":"ContainerDied","Data":"1d68d4d70c52e1eb835976cd711066f937d35de531010f04b70f856cf7b4beb7"}
Sep 11 17:56:40.898579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:40.898179    2563 scope.go:117] "RemoveContainer" containerID="51321b0923c7478e1c988bc5ed19d0a8a148f32cfd48f4ab04510a2f568f596f"
Sep 11 17:56:40.956462 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:40.956395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:56:41.120382 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-socket\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120414    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-data\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-istio-token\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-certs\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a231928-c913-4146-84dd-955e6a4ab759-istio-podinfo\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120604 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-envoy\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120876 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-credential-socket\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120876 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120655    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c669\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-kube-api-access-6c669\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120876 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a231928-c913-4146-84dd-955e6a4ab759-istiod-ca-cert\") pod \"4a231928-c913-4146-84dd-955e6a4ab759\" (UID: \"4a231928-c913-4146-84dd-955e6a4ab759\") "
Sep 11 17:56:41.120876 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120768    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:56:41.120876 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:56:41.121104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:56:41.121104 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.120958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:56:41.121201 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.121117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a231928-c913-4146-84dd-955e6a4ab759-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:41.121201 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.121171    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.121201 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.121191    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.121355 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.121208    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.121355 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.121223    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.122994 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.122960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:56:41.123237 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.123204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-kube-api-access-6c669" (OuterVolumeSpecName: "kube-api-access-6c669") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "kube-api-access-6c669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:41.123369 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.123354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:41.123480 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.123461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a231928-c913-4146-84dd-955e6a4ab759-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a231928-c913-4146-84dd-955e6a4ab759" (UID: "4a231928-c913-4146-84dd-955e6a4ab759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:56:41.222830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.222735    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a231928-c913-4146-84dd-955e6a4ab759-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.222830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.222775    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a231928-c913-4146-84dd-955e6a4ab759-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.222830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.222786    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c669\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-kube-api-access-6c669\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.222830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.222796    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a231928-c913-4146-84dd-955e6a4ab759-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.222830 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.222804    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a231928-c913-4146-84dd-955e6a4ab759-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:41.858761 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.858482    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"]
Sep 11 17:56:41.858955 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.858755    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27" podUID="b53598b2-74b9-4d44-b134-ebf177756004" containerName="payload-processing" containerID="cri-o://b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009" gracePeriod=30
Sep 11 17:56:41.907626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.907062    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"
Sep 11 17:56:41.907626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.907272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf" event={"ID":"4a231928-c913-4146-84dd-955e6a4ab759","Type":"ContainerDied","Data":"49bb0a2fa153fc53dc4832af04b2bc2218f0b867cbd71d207dcf6f1395ae3db8"}
Sep 11 17:56:41.907626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.907373    2563 scope.go:117] "RemoveContainer" containerID="1d68d4d70c52e1eb835976cd711066f937d35de531010f04b70f856cf7b4beb7"
Sep 11 17:56:41.938377 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.938343    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"]
Sep 11 17:56:41.941131 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:41.941102    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fcdaf27d-openshift-default-5746c8b748-ndpcf"]
Sep 11 17:56:42.116566 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.116541    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:56:42.233219 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.233178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn2g2\" (UniqueName: \"kubernetes.io/projected/b53598b2-74b9-4d44-b134-ebf177756004-kube-api-access-kn2g2\") pod \"b53598b2-74b9-4d44-b134-ebf177756004\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") "
Sep 11 17:56:42.233417 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.233280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53598b2-74b9-4d44-b134-ebf177756004-plugins-config-volume\") pod \"b53598b2-74b9-4d44-b134-ebf177756004\" (UID: \"b53598b2-74b9-4d44-b134-ebf177756004\") "
Sep 11 17:56:42.233702 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.233677    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53598b2-74b9-4d44-b134-ebf177756004-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b53598b2-74b9-4d44-b134-ebf177756004" (UID: "b53598b2-74b9-4d44-b134-ebf177756004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:42.235529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.235507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53598b2-74b9-4d44-b134-ebf177756004-kube-api-access-kn2g2" (OuterVolumeSpecName: "kube-api-access-kn2g2") pod "b53598b2-74b9-4d44-b134-ebf177756004" (UID: "b53598b2-74b9-4d44-b134-ebf177756004"). InnerVolumeSpecName "kube-api-access-kn2g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:42.334895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.334847    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn2g2\" (UniqueName: \"kubernetes.io/projected/b53598b2-74b9-4d44-b134-ebf177756004-kube-api-access-kn2g2\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:42.334895 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.334884    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b53598b2-74b9-4d44-b134-ebf177756004-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:56:42.915291 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.915259    2563 generic.go:356] "Generic (PLEG): container finished" podID="b53598b2-74b9-4d44-b134-ebf177756004" containerID="b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009" exitCode=0
Sep 11 17:56:42.915728 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.915301    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"
Sep 11 17:56:42.915728 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.915323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27" event={"ID":"b53598b2-74b9-4d44-b134-ebf177756004","Type":"ContainerDied","Data":"b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009"}
Sep 11 17:56:42.915728 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.915364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27" event={"ID":"b53598b2-74b9-4d44-b134-ebf177756004","Type":"ContainerDied","Data":"764eae297b2a72c2c9a85b92cabc73a52cdde4877554d10cf7ca18e6c5d34cf1"}
Sep 11 17:56:42.915728 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.915386    2563 scope.go:117] "RemoveContainer" containerID="b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009"
Sep 11 17:56:42.925964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.925945    2563 scope.go:117] "RemoveContainer" containerID="b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009"
Sep 11 17:56:42.926267 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:56:42.926245    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009\": container with ID starting with b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009 not found: ID does not exist" containerID="b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009"
Sep 11 17:56:42.926352 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.926279    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009"} err="failed to get container status \"b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009\": rpc error: code = NotFound desc = could not find container \"b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009\": container with ID starting with b0eae529abfb1cfd3025a11d5009ddaf024dbff25319e9462dd56a39f98be009 not found: ID does not exist"
Sep 11 17:56:42.939263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.939236    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"]
Sep 11 17:56:42.946314 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:42.946288    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-104872-8546c44b65-mbr27"]
Sep 11 17:56:43.782379 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:43.782345    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a231928-c913-4146-84dd-955e6a4ab759" path="/var/lib/kubelet/pods/4a231928-c913-4146-84dd-955e6a4ab759/volumes"
Sep 11 17:56:43.782775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:43.782762    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53598b2-74b9-4d44-b134-ebf177756004" path="/var/lib/kubelet/pods/b53598b2-74b9-4d44-b134-ebf177756004/volumes"
Sep 11 17:56:50.247924 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.247843    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"]
Sep 11 17:56:50.248587 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35c91276-1dcf-45e2-8a20-a0183d96f5d3" containerName="maas-api"
Sep 11 17:56:50.248660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248591    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="35c91276-1dcf-45e2-8a20-a0183d96f5d3" containerName="maas-api"
Sep 11 17:56:50.248660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248616    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy"
Sep 11 17:56:50.248660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248626    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy"
Sep 11 17:56:50.248660 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248654    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248663    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248672    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53598b2-74b9-4d44-b134-ebf177756004" containerName="payload-processing"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248682    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53598b2-74b9-4d44-b134-ebf177756004" containerName="payload-processing"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248772    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53598b2-74b9-4d44-b134-ebf177756004" containerName="payload-processing"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248780    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248786    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="35c91276-1dcf-45e2-8a20-a0183d96f5d3" containerName="maas-api"
Sep 11 17:56:50.248870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.248791    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a231928-c913-4146-84dd-955e6a4ab759" containerName="istio-proxy"
Sep 11 17:56:50.251911 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.251897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.255871 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.255844    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-6baa6b-openshift-default-dockercfg-vz4jt\""
Sep 11 17:56:50.279354 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.279325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"]
Sep 11 17:56:50.313991 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.313956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-credential-socket\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.313996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bxv\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-kube-api-access-22bxv\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istiod-ca-cert\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-certs\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-data\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-podinfo\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314262 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-socket\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-token\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.314579 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.314356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-envoy\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.414957 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.414909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-credential-socket\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415172 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.414963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bxv\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-kube-api-access-22bxv\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415172 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istiod-ca-cert\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415172 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-certs\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415172 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-data\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415172 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-podinfo\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415445 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-socket\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415445 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-token\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415445 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-envoy\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415445 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-credential-socket\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415689 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-data\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415689 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-certs\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415807 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-socket\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.415870 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.415793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istiod-ca-cert\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.417629 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.417606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-envoy\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.417767 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.417727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-podinfo\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.434773 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.434745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-token\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.435229 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.435208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bxv\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-kube-api-access-22bxv\") pod \"e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") " pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.563661 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.563563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:50.736055 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.736007    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"]
Sep 11 17:56:50.740216 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.740196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.743169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.743143    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-21e472-openshift-default-dockercfg-lj42p\""
Sep 11 17:56:50.751970 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.751939    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"]
Sep 11 17:56:50.819441 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-data\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-credential-socket\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-podinfo\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8nxv\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-kube-api-access-l8nxv\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-certs\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819844 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-token\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819844 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-socket\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819844 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-envoy\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.819844 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.819777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istiod-ca-cert\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.857100 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.857072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"]
Sep 11 17:56:50.858251 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:56:50.858225    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32a26e1_0037_4122_9b7c_e5b3164ef34d.slice/crio-9474078f032ebf406f282a91dd768e513a6a6ae0f3fd27f1dbef08cab62ce5a4 WatchSource:0}: Error finding container 9474078f032ebf406f282a91dd768e513a6a6ae0f3fd27f1dbef08cab62ce5a4: Status 404 returned error can't find the container with id 9474078f032ebf406f282a91dd768e513a6a6ae0f3fd27f1dbef08cab62ce5a4
Sep 11 17:56:50.860594 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.860562    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:50.860690 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.860642    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:50.860690 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.860677    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:50.920639 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.920610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8nxv\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-kube-api-access-l8nxv\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.920753 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.920679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-certs\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.920866 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.920844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-token\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.920930 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.920903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-socket\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.920982 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.920954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-envoy\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921036 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.920986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-certs\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921036 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istiod-ca-cert\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921140 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-data\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921140 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-credential-socket\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921244 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-podinfo\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921244 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-socket\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921446 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-data\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921545 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-credential-socket\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.921948 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.921926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istiod-ca-cert\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.923384 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.923365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-envoy\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.923914 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.923898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-podinfo\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.942568 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.942540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-token\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.948430 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.948405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8nxv\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-kube-api-access-l8nxv\") pod \"e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") " pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:50.967460 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.967419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm" event={"ID":"f32a26e1-0037-4122-9b7c-e5b3164ef34d","Type":"ContainerStarted","Data":"0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e"}
Sep 11 17:56:50.967666 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:50.967481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm" event={"ID":"f32a26e1-0037-4122-9b7c-e5b3164ef34d","Type":"ContainerStarted","Data":"9474078f032ebf406f282a91dd768e513a6a6ae0f3fd27f1dbef08cab62ce5a4"}
Sep 11 17:56:51.020921 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.020843    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm" podStartSLOduration=1.020819297 podStartE2EDuration="1.020819297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:51.018075273 +0000 UTC m=+1077.781530001" watchObservedRunningTime="2026-09-11 17:56:51.020819297 +0000 UTC m=+1077.784274028"
Sep 11 17:56:51.053567 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.053304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:51.347746 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.347713    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"]
Sep 11 17:56:51.348852 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:56:51.348807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00c8a41_8b2c_46a3_a233_44a1a199aa79.slice/crio-60a7b0f60e4ee77cdd6041124dad06d2f12050a85a6b868f0c0d531ff126c734 WatchSource:0}: Error finding container 60a7b0f60e4ee77cdd6041124dad06d2f12050a85a6b868f0c0d531ff126c734: Status 404 returned error can't find the container with id 60a7b0f60e4ee77cdd6041124dad06d2f12050a85a6b868f0c0d531ff126c734
Sep 11 17:56:51.351265 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.351230    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:51.351332 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.351313    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:51.351384 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.351361    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:56:51.563934 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.563900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:51.569723 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.569692    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:51.975096 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.975060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" event={"ID":"e00c8a41-8b2c-46a3-a233-44a1a199aa79","Type":"ContainerStarted","Data":"d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"}
Sep 11 17:56:51.975269 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.975127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" event={"ID":"e00c8a41-8b2c-46a3-a233-44a1a199aa79","Type":"ContainerStarted","Data":"60a7b0f60e4ee77cdd6041124dad06d2f12050a85a6b868f0c0d531ff126c734"}
Sep 11 17:56:51.975500 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.975465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:51.976605 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.976582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:56:51.996759 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:51.996708    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podStartSLOduration=1.9966905000000001 podStartE2EDuration="1.9966905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:51.994770809 +0000 UTC m=+1078.758225548" watchObservedRunningTime="2026-09-11 17:56:51.9966905 +0000 UTC m=+1078.760145226"
Sep 11 17:56:52.054339 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:52.054303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:52.059417 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:52.059391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:52.982797 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:52.982769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:52.983688 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:52.983668    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:56:57.866423 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.866384    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"]
Sep 11 17:56:57.871457 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.871438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:57.874434 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.874416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-21e472\""
Sep 11 17:56:57.874655 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.874635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-21e472-dockercfg-tqc2f\""
Sep 11 17:56:57.883948 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.883923    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"]
Sep 11 17:56:57.894942 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.894883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:57.894942 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.894934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkk7n\" (UniqueName: \"kubernetes.io/projected/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-kube-api-access-bkk7n\") pod \"payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:57.998516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.996653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:57.998516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.996736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkk7n\" (UniqueName: \"kubernetes.io/projected/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-kube-api-access-bkk7n\") pod \"payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:57.998516 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:57.998047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:58.019933 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:58.019902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkk7n\" (UniqueName: \"kubernetes.io/projected/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-kube-api-access-bkk7n\") pod \"payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:58.183599 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:58.183464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:58.464527 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:58.464457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"]
Sep 11 17:56:58.465550 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:56:58.465522    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67faa52_d8cf_4a89_acff_89d3bc17a7f8.slice/crio-40de19714e6a0ed828baab3251e02f1d6045cd74a7f1d2615869833db1178ad5 WatchSource:0}: Error finding container 40de19714e6a0ed828baab3251e02f1d6045cd74a7f1d2615869833db1178ad5: Status 404 returned error can't find the container with id 40de19714e6a0ed828baab3251e02f1d6045cd74a7f1d2615869833db1178ad5
Sep 11 17:56:59.029882 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:59.029845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4" event={"ID":"c67faa52-d8cf-4a89-acff-89d3bc17a7f8","Type":"ContainerStarted","Data":"bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d"}
Sep 11 17:56:59.030278 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:59.029912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:56:59.030278 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:59.029935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4" event={"ID":"c67faa52-d8cf-4a89-acff-89d3bc17a7f8","Type":"ContainerStarted","Data":"40de19714e6a0ed828baab3251e02f1d6045cd74a7f1d2615869833db1178ad5"}
Sep 11 17:56:59.048985 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:56:59.048933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4" podStartSLOduration=2.048913116 podStartE2EDuration="2.048913116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:59.046444153 +0000 UTC m=+1085.809898879" watchObservedRunningTime="2026-09-11 17:56:59.048913116 +0000 UTC m=+1085.812367843"
Sep 11 17:57:10.038648 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:10.038604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 17:57:24.024736 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:24.024680    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 17:57:24.025214 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:24.024798    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 17:57:26.196031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:26.195920    2563 generic.go:356] "Generic (PLEG): container finished" podID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerID="d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6" exitCode=0
Sep 11 17:57:26.196031 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:26.195988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" event={"ID":"e00c8a41-8b2c-46a3-a233-44a1a199aa79","Type":"ContainerDied","Data":"d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"}
Sep 11 17:57:26.196463 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:26.196451    2563 scope.go:117] "RemoveContainer" containerID="d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"
Sep 11 17:57:26.197672 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:26.197634    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:57:26.197779 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:26.197707    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:57:26.197779 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:26.197745    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:57:27.210189 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:27.210155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" event={"ID":"e00c8a41-8b2c-46a3-a233-44a1a199aa79","Type":"ContainerStarted","Data":"86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4"}
Sep 11 17:57:28.054057 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:28.054017    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:57:28.054263 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:28.054237    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:57:28.054315 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:28.054283    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 17:57:29.054352 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:29.054318    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:57:29.054865 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:29.054399    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 17:57:30.054205 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:30.054171    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:57:30.054401 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:30.054226    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 17:57:31.054149 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:31.054107    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:57:31.054323 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:31.054167    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 17:57:32.054348 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:32.054299    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:57:32.054883 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:32.054369    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 17:57:32.893839 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:32.893796    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"]
Sep 11 17:57:32.897850 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:32.897829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:32.900882 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:32.900852    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-7f85c3-serving-cert\""
Sep 11 17:57:32.915399 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:32.915365    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"]
Sep 11 17:57:33.013221 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.013181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fee485f-34a7-46a7-9a8e-8147773ae477-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.013221 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.013222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.013464 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.013305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.013464 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.013341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665tt\" (UniqueName: \"kubernetes.io/projected/7fee485f-34a7-46a7-9a8e-8147773ae477-kube-api-access-665tt\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.058245 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.058213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:57:33.058721 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.058524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:57:33.059209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.059192    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 17:57:33.114662 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.114626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fee485f-34a7-46a7-9a8e-8147773ae477-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.114873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.114685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.114873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.114795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.114873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.114854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665tt\" (UniqueName: \"kubernetes.io/projected/7fee485f-34a7-46a7-9a8e-8147773ae477-kube-api-access-665tt\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.115048 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:57:33.114900    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-serving-cert: secret "maas-api-e2e-shared-b-w2-7f85c3-serving-cert" not found
Sep 11 17:57:33.115048 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:57:33.114993    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls podName:7fee485f-34a7-46a7-9a8e-8147773ae477 nodeName:}" failed. No retries permitted until 2026-09-11 17:57:33.614968098 +0000 UTC m=+1120.378422813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls") pod "maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" (UID: "7fee485f-34a7-46a7-9a8e-8147773ae477") : secret "maas-api-e2e-shared-b-w2-7f85c3-serving-cert" not found
Sep 11 17:57:33.115476 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.115453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fee485f-34a7-46a7-9a8e-8147773ae477-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.117609 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.117582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.135151 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.135119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-665tt\" (UniqueName: \"kubernetes.io/projected/7fee485f-34a7-46a7-9a8e-8147773ae477-kube-api-access-665tt\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.620379 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.620340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.623209 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.623179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:33.811898 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:33.811854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:34.091669 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:34.091640    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"]
Sep 11 17:57:34.092177 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:57:34.092153    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fee485f_34a7_46a7_9a8e_8147773ae477.slice/crio-73c50e8217c155e58de4c07d6d106a0e13a7e675353b8efa00a3d197e4d242cf WatchSource:0}: Error finding container 73c50e8217c155e58de4c07d6d106a0e13a7e675353b8efa00a3d197e4d242cf: Status 404 returned error can't find the container with id 73c50e8217c155e58de4c07d6d106a0e13a7e675353b8efa00a3d197e4d242cf
Sep 11 17:57:34.093826 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:34.093809    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:57:34.254780 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:34.254746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" event={"ID":"7fee485f-34a7-46a7-9a8e-8147773ae477","Type":"ContainerStarted","Data":"73c50e8217c155e58de4c07d6d106a0e13a7e675353b8efa00a3d197e4d242cf"}
Sep 11 17:57:35.261971 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:35.261934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" event={"ID":"7fee485f-34a7-46a7-9a8e-8147773ae477","Type":"ContainerStarted","Data":"8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358"}
Sep 11 17:57:35.262410 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:35.262153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:35.281337 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:35.281279    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" podStartSLOduration=2.996135627 podStartE2EDuration="3.281263053s" totalImagesPullingTime="285.127426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:57:35.278870782 +0000 UTC m=+1122.042325509" watchObservedRunningTime="2026-09-11 17:57:35.281263053 +0000 UTC m=+1122.044717897"
Sep 11 17:57:41.273927 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:41.273896    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 17:57:56.875762 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:56.875730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"]
Sep 11 17:57:56.876307 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:56.875954    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc" podUID="d1649022-7039-4c08-8dcd-a2b9b3a169d5" containerName="maas-api" containerID="cri-o://baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1" gracePeriod=30
Sep 11 17:57:57.137593 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.137475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:57:57.276993 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.276957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1649022-7039-4c08-8dcd-a2b9b3a169d5-openshift-service-ca\") pod \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") "
Sep 11 17:57:57.277385 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.277362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-527t9\" (UniqueName: \"kubernetes.io/projected/d1649022-7039-4c08-8dcd-a2b9b3a169d5-kube-api-access-527t9\") pod \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") "
Sep 11 17:57:57.277668 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.277649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-metrics-certs\") pod \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") "
Sep 11 17:57:57.277827 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.277812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls\") pod \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\" (UID: \"d1649022-7039-4c08-8dcd-a2b9b3a169d5\") "
Sep 11 17:57:57.279179 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.279135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1649022-7039-4c08-8dcd-a2b9b3a169d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1649022-7039-4c08-8dcd-a2b9b3a169d5" (UID: "d1649022-7039-4c08-8dcd-a2b9b3a169d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:57.284013 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.283969    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d1649022-7039-4c08-8dcd-a2b9b3a169d5" (UID: "d1649022-7039-4c08-8dcd-a2b9b3a169d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:57.299508 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.296735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1649022-7039-4c08-8dcd-a2b9b3a169d5" (UID: "d1649022-7039-4c08-8dcd-a2b9b3a169d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:57.299508 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.297344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1649022-7039-4c08-8dcd-a2b9b3a169d5-kube-api-access-527t9" (OuterVolumeSpecName: "kube-api-access-527t9") pod "d1649022-7039-4c08-8dcd-a2b9b3a169d5" (UID: "d1649022-7039-4c08-8dcd-a2b9b3a169d5"). InnerVolumeSpecName "kube-api-access-527t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:57:57.379980 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.379939    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1649022-7039-4c08-8dcd-a2b9b3a169d5-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:57:57.380168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.379985    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527t9\" (UniqueName: \"kubernetes.io/projected/d1649022-7039-4c08-8dcd-a2b9b3a169d5-kube-api-access-527t9\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:57:57.380168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.380004    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:57:57.380168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.380026    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1649022-7039-4c08-8dcd-a2b9b3a169d5-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:57:57.396214 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.396128    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1649022-7039-4c08-8dcd-a2b9b3a169d5" containerID="baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1" exitCode=0
Sep 11 17:57:57.396214 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.396185    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"
Sep 11 17:57:57.396388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.396214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc" event={"ID":"d1649022-7039-4c08-8dcd-a2b9b3a169d5","Type":"ContainerDied","Data":"baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1"}
Sep 11 17:57:57.396388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.396283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc" event={"ID":"d1649022-7039-4c08-8dcd-a2b9b3a169d5","Type":"ContainerDied","Data":"c4657c756ff38dbad53507a67338d42e489a35d8fbb233ba5d2eecec5bf9285f"}
Sep 11 17:57:57.396388 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.396314    2563 scope.go:117] "RemoveContainer" containerID="baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1"
Sep 11 17:57:57.409145 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.409125    2563 scope.go:117] "RemoveContainer" containerID="baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1"
Sep 11 17:57:57.409543 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:57:57.409522    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1\": container with ID starting with baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1 not found: ID does not exist" containerID="baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1"
Sep 11 17:57:57.409614 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.409556    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1"} err="failed to get container status \"baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1\": rpc error: code = NotFound desc = could not find container \"baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1\": container with ID starting with baf7100b22552769d7e8ae248a4825cb0bcc20e440854763c2dc4992b14c08a1 not found: ID does not exist"
Sep 11 17:57:57.429930 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.429879    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"]
Sep 11 17:57:57.433459 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.433433    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-715253-7656949976-g87jc"]
Sep 11 17:57:57.782632 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:57:57.782548    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1649022-7039-4c08-8dcd-a2b9b3a169d5" path="/var/lib/kubelet/pods/d1649022-7039-4c08-8dcd-a2b9b3a169d5/volumes"
Sep 11 17:58:04.563453 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:04.563415    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"]
Sep 11 17:58:04.563940 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:04.563685    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy" containerID="cri-o://66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a" gracePeriod=30
Sep 11 17:58:09.832819 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.832750    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:58:09.912258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-credential-socket\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-token\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-podinfo\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1af3b5-3380-46da-b3f4-d332e02c61da-istiod-ca-cert\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-socket\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:09.912696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-envoy\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-certs\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qvbt\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-kube-api-access-9qvbt\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-data\") pod \"cd1af3b5-3380-46da-b3f4-d332e02c61da\" (UID: \"cd1af3b5-3380-46da-b3f4-d332e02c61da\") "
Sep 11 17:58:09.912947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:09.912947 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1af3b5-3380-46da-b3f4-d332e02c61da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:58:09.913046 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.912955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:09.913110 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.913092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:09.913202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.913182    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:09.913202 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.913199    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd1af3b5-3380-46da-b3f4-d332e02c61da-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:09.913302 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.913208    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:09.913302 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.913220    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:09.915183 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.915147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:09.915316 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.915292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:09.915472 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.915446    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-kube-api-access-9qvbt" (OuterVolumeSpecName: "kube-api-access-9qvbt") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "kube-api-access-9qvbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:09.915540 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:09.915480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd1af3b5-3380-46da-b3f4-d332e02c61da" (UID: "cd1af3b5-3380-46da-b3f4-d332e02c61da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:58:10.013999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.013963    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:10.013999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.013993    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvbt\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-kube-api-access-9qvbt\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:10.013999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.014004    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:10.014240 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.014013    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:10.014240 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.014033    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd1af3b5-3380-46da-b3f4-d332e02c61da-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:10.040396 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.040322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"]
Sep 11 17:58:10.040893 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.040874    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1649022-7039-4c08-8dcd-a2b9b3a169d5" containerName="maas-api"
Sep 11 17:58:10.041003 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.040896    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1649022-7039-4c08-8dcd-a2b9b3a169d5" containerName="maas-api"
Sep 11 17:58:10.041003 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.040916    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy"
Sep 11 17:58:10.041003 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.040925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy"
Sep 11 17:58:10.041206 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.041044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1649022-7039-4c08-8dcd-a2b9b3a169d5" containerName="maas-api"
Sep 11 17:58:10.041206 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.041064    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy"
Sep 11 17:58:10.041206 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.041073    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy"
Sep 11 17:58:10.045096 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.045074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.047846 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.047824    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ed18394d-serving-cert\""
Sep 11 17:58:10.057321 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.057291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"]
Sep 11 17:58:10.115199 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.115104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c5d68b-f884-4487-8a6c-65bfa1564016-openshift-service-ca\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.115199 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.115167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.115390 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.115259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkbb\" (UniqueName: \"kubernetes.io/projected/b3c5d68b-f884-4487-8a6c-65bfa1564016-kube-api-access-9dkbb\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.115390 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.115297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-metrics-certs\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.216851 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.216818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c5d68b-f884-4487-8a6c-65bfa1564016-openshift-service-ca\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.217047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.216877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.217047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.216900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkbb\" (UniqueName: \"kubernetes.io/projected/b3c5d68b-f884-4487-8a6c-65bfa1564016-kube-api-access-9dkbb\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.217047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.216918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-metrics-certs\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.217047 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:10.217039    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-serving-cert: secret "maas-api-e2e-mt-ed18394d-serving-cert" not found
Sep 11 17:58:10.217255 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:10.217101    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls podName:b3c5d68b-f884-4487-8a6c-65bfa1564016 nodeName:}" failed. No retries permitted until 2026-09-11 17:58:10.717085058 +0000 UTC m=+1157.480539766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls") pod "maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" (UID: "b3c5d68b-f884-4487-8a6c-65bfa1564016") : secret "maas-api-e2e-mt-ed18394d-serving-cert" not found
Sep 11 17:58:10.217443 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.217423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c5d68b-f884-4487-8a6c-65bfa1564016-openshift-service-ca\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.219434 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.219416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-metrics-certs\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.227619 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.227592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkbb\" (UniqueName: \"kubernetes.io/projected/b3c5d68b-f884-4487-8a6c-65bfa1564016-kube-api-access-9dkbb\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.478151 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.478045    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerID="66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a" exitCode=0
Sep 11 17:58:10.478151 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.478118    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"
Sep 11 17:58:10.478391 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.478129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" event={"ID":"cd1af3b5-3380-46da-b3f4-d332e02c61da","Type":"ContainerDied","Data":"66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a"}
Sep 11 17:58:10.478391 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.478275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt" event={"ID":"cd1af3b5-3380-46da-b3f4-d332e02c61da","Type":"ContainerDied","Data":"25bd4adc4284fdcb182b8e8602debdf03b1c01d311f9b22447c0313278be6711"}
Sep 11 17:58:10.478391 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.478311    2563 scope.go:117] "RemoveContainer" containerID="66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a"
Sep 11 17:58:10.488708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.488688    2563 scope.go:117] "RemoveContainer" containerID="be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"
Sep 11 17:58:10.499033 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.499007    2563 scope.go:117] "RemoveContainer" containerID="66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a"
Sep 11 17:58:10.499418 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:10.499393    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a\": container with ID starting with 66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a not found: ID does not exist" containerID="66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a"
Sep 11 17:58:10.499618 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.499428    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a"} err="failed to get container status \"66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a\": rpc error: code = NotFound desc = could not find container \"66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a\": container with ID starting with 66b22f9d21b542917e74e205c28fef2485bb9dea7d260e3554b7ea4ca8646b7a not found: ID does not exist"
Sep 11 17:58:10.499618 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.499454    2563 scope.go:117] "RemoveContainer" containerID="be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"
Sep 11 17:58:10.499821 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:10.499803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741\": container with ID starting with be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741 not found: ID does not exist" containerID="be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"
Sep 11 17:58:10.499885 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.499830    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741"} err="failed to get container status \"be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741\": rpc error: code = NotFound desc = could not find container \"be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741\": container with ID starting with be2fffb58b2fe2788ff107f80ed6dae7cebf38931f7c6e28c57941959e764741 not found: ID does not exist"
Sep 11 17:58:10.507158 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.507127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"]
Sep 11 17:58:10.513779 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.513752    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-715253-openshift-default-869fbc9f76-hghvt"]
Sep 11 17:58:10.655450 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.655418    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"]
Sep 11 17:58:10.655936 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.655919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy"
Sep 11 17:58:10.656011 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.655940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" containerName="istio-proxy"
Sep 11 17:58:10.660442 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.660427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.663850 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.663831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ed18394d-dockercfg-2hd7l\""
Sep 11 17:58:10.663983 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.663910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ed18394d\""
Sep 11 17:58:10.672224 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.672198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"]
Sep 11 17:58:10.676010 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.675984    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"]
Sep 11 17:58:10.683171 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.683140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.700542 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.700514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"]
Sep 11 17:58:10.722899 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.722866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwd2r\" (UniqueName: \"kubernetes.io/projected/895dcca2-8198-43f0-b57b-f013ddf83189-kube-api-access-pwd2r\") pod \"payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") " pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.723064 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.722909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.723064 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.723008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddffs\" (UniqueName: \"kubernetes.io/projected/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-kube-api-access-ddffs\") pod \"payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.723064 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.723048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.723171 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.723081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895dcca2-8198-43f0-b57b-f013ddf83189-plugins-config-volume\") pod \"payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") " pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.725573 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.725545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls\") pod \"maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.824047 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.824001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwd2r\" (UniqueName: \"kubernetes.io/projected/895dcca2-8198-43f0-b57b-f013ddf83189-kube-api-access-pwd2r\") pod \"payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") " pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.824231 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.824056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.824231 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.824119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddffs\" (UniqueName: \"kubernetes.io/projected/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-kube-api-access-ddffs\") pod \"payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.824231 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.824157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895dcca2-8198-43f0-b57b-f013ddf83189-plugins-config-volume\") pod \"payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") " pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.824909 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.824891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895dcca2-8198-43f0-b57b-f013ddf83189-plugins-config-volume\") pod \"payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") " pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.824960 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.824893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.832959 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.832936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwd2r\" (UniqueName: \"kubernetes.io/projected/895dcca2-8198-43f0-b57b-f013ddf83189-kube-api-access-pwd2r\") pod \"payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") " pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:10.833278 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.832986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddffs\" (UniqueName: \"kubernetes.io/projected/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-kube-api-access-ddffs\") pod \"payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.959573 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.959424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:10.972680 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.972653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:10.994636 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:10.994607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:11.243118 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.243090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"]
Sep 11 17:58:11.244100 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:58:11.244074    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c5d68b_f884_4487_8a6c_65bfa1564016.slice/crio-a2d77fad31bcc0fa6a26afb046f942f6884c2dd82dc4ef3e7c0e8bd5cf1efded WatchSource:0}: Error finding container a2d77fad31bcc0fa6a26afb046f942f6884c2dd82dc4ef3e7c0e8bd5cf1efded: Status 404 returned error can't find the container with id a2d77fad31bcc0fa6a26afb046f942f6884c2dd82dc4ef3e7c0e8bd5cf1efded
Sep 11 17:58:11.267873 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.267813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"]
Sep 11 17:58:11.268405 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:58:11.268380    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ff55e3_5b4c_42f0_b5cd_564ccc7ff187.slice/crio-ef916ac97b5dd4d778188b4af5eb555b4b93560a111cf442533aac3877b3ca1e WatchSource:0}: Error finding container ef916ac97b5dd4d778188b4af5eb555b4b93560a111cf442533aac3877b3ca1e: Status 404 returned error can't find the container with id ef916ac97b5dd4d778188b4af5eb555b4b93560a111cf442533aac3877b3ca1e
Sep 11 17:58:11.293967 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.293941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"]
Sep 11 17:58:11.309713 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:58:11.309685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895dcca2_8198_43f0_b57b_f013ddf83189.slice/crio-5c6efbb1c5ac02e1a2b030206979ab3211d7426fcfbfcfef28a05587eced9ac9 WatchSource:0}: Error finding container 5c6efbb1c5ac02e1a2b030206979ab3211d7426fcfbfcfef28a05587eced9ac9: Status 404 returned error can't find the container with id 5c6efbb1c5ac02e1a2b030206979ab3211d7426fcfbfcfef28a05587eced9ac9
Sep 11 17:58:11.497403 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.497363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn" event={"ID":"895dcca2-8198-43f0-b57b-f013ddf83189","Type":"ContainerStarted","Data":"4c40ea766e501796cf1228f9c17fb9d279d60aa798bbb74cb6576fbbdfcd6329"}
Sep 11 17:58:11.497643 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.497467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn" event={"ID":"895dcca2-8198-43f0-b57b-f013ddf83189","Type":"ContainerStarted","Data":"5c6efbb1c5ac02e1a2b030206979ab3211d7426fcfbfcfef28a05587eced9ac9"}
Sep 11 17:58:11.497643 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.497534    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:11.499142 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.499110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw" event={"ID":"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187","Type":"ContainerStarted","Data":"7ad338c323e5375af3e5b5c214310bc396603c4a449bda7420d14dad1b3a5e02"}
Sep 11 17:58:11.499279 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.499166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw" event={"ID":"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187","Type":"ContainerStarted","Data":"ef916ac97b5dd4d778188b4af5eb555b4b93560a111cf442533aac3877b3ca1e"}
Sep 11 17:58:11.499279 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.499194    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:11.500616 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.500592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" event={"ID":"b3c5d68b-f884-4487-8a6c-65bfa1564016","Type":"ContainerStarted","Data":"a2d77fad31bcc0fa6a26afb046f942f6884c2dd82dc4ef3e7c0e8bd5cf1efded"}
Sep 11 17:58:11.517557 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.517508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn" podStartSLOduration=1.517479346 podStartE2EDuration="1.517479346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:11.515793858 +0000 UTC m=+1158.279248586" watchObservedRunningTime="2026-09-11 17:58:11.517479346 +0000 UTC m=+1158.280934134"
Sep 11 17:58:11.536606 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.536556    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw" podStartSLOduration=1.5365385 podStartE2EDuration="1.5365385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:11.534121217 +0000 UTC m=+1158.297575956" watchObservedRunningTime="2026-09-11 17:58:11.5365385 +0000 UTC m=+1158.299993226"
Sep 11 17:58:11.783932 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:11.783844    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd1af3b5-3380-46da-b3f4-d332e02c61da" path="/var/lib/kubelet/pods/cd1af3b5-3380-46da-b3f4-d332e02c61da/volumes"
Sep 11 17:58:12.509368 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:12.509329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" event={"ID":"b3c5d68b-f884-4487-8a6c-65bfa1564016","Type":"ContainerStarted","Data":"883ee53438dca120fd642966d10afc6a82a1379888cedd50e078f0acf4c51f46"}
Sep 11 17:58:12.509857 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:12.509794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:12.529432 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:12.529368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" podStartSLOduration=1.973357836 podStartE2EDuration="2.529350347s" totalImagesPullingTime="555.992511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:12.526685726 +0000 UTC m=+1159.290140484" watchObservedRunningTime="2026-09-11 17:58:12.529350347 +0000 UTC m=+1159.292805074"
Sep 11 17:58:14.552519 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:14.552401    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"]
Sep 11 17:58:14.552919 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:14.552620    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm" podUID="f32a26e1-0037-4122-9b7c-e5b3164ef34d" containerName="istio-proxy" containerID="cri-o://0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e" gracePeriod=30
Sep 11 17:58:16.235420 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.235382    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"]
Sep 11 17:58:16.235907 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.235616    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p" podUID="a4ee9138-0121-4b07-a24f-2b86c4ea10af" containerName="maas-api" containerID="cri-o://7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075" gracePeriod=30
Sep 11 17:58:16.503271 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.503206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:58:16.538907 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.538872    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4ee9138-0121-4b07-a24f-2b86c4ea10af" containerID="7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075" exitCode=0
Sep 11 17:58:16.539117 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.538923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"
Sep 11 17:58:16.539117 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.538936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p" event={"ID":"a4ee9138-0121-4b07-a24f-2b86c4ea10af","Type":"ContainerDied","Data":"7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075"}
Sep 11 17:58:16.539117 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.539004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p" event={"ID":"a4ee9138-0121-4b07-a24f-2b86c4ea10af","Type":"ContainerDied","Data":"d2b675f54c326fb700c012c27fcc302166749c1185cb017f3b8e37a6bbcad506"}
Sep 11 17:58:16.539117 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.539026    2563 scope.go:117] "RemoveContainer" containerID="7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075"
Sep 11 17:58:16.548630 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.548610    2563 scope.go:117] "RemoveContainer" containerID="7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075"
Sep 11 17:58:16.548928 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:16.548906    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075\": container with ID starting with 7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075 not found: ID does not exist" containerID="7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075"
Sep 11 17:58:16.548998 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.548941    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075"} err="failed to get container status \"7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075\": rpc error: code = NotFound desc = could not find container \"7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075\": container with ID starting with 7dc1c86bc619b24678a9d0391bffe52498efe88386bfbdea695bb64c3a3b4075 not found: ID does not exist"
Sep 11 17:58:16.587092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.587045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6f9t\" (UniqueName: \"kubernetes.io/projected/a4ee9138-0121-4b07-a24f-2b86c4ea10af-kube-api-access-q6f9t\") pod \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") "
Sep 11 17:58:16.587286 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.587098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-metrics-certs\") pod \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") "
Sep 11 17:58:16.587286 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.587228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ee9138-0121-4b07-a24f-2b86c4ea10af-openshift-service-ca\") pod \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") "
Sep 11 17:58:16.587407 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.587350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-maas-api-tls\") pod \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\" (UID: \"a4ee9138-0121-4b07-a24f-2b86c4ea10af\") "
Sep 11 17:58:16.587642 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.587616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ee9138-0121-4b07-a24f-2b86c4ea10af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4ee9138-0121-4b07-a24f-2b86c4ea10af" (UID: "a4ee9138-0121-4b07-a24f-2b86c4ea10af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:58:16.587794 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.587777    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ee9138-0121-4b07-a24f-2b86c4ea10af-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:16.589323 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.589275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4ee9138-0121-4b07-a24f-2b86c4ea10af" (UID: "a4ee9138-0121-4b07-a24f-2b86c4ea10af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:58:16.589433 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.589356    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ee9138-0121-4b07-a24f-2b86c4ea10af-kube-api-access-q6f9t" (OuterVolumeSpecName: "kube-api-access-q6f9t") pod "a4ee9138-0121-4b07-a24f-2b86c4ea10af" (UID: "a4ee9138-0121-4b07-a24f-2b86c4ea10af"). InnerVolumeSpecName "kube-api-access-q6f9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:16.589678 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.589658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4ee9138-0121-4b07-a24f-2b86c4ea10af" (UID: "a4ee9138-0121-4b07-a24f-2b86c4ea10af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:58:16.689399 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.689207    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:16.689399 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.689246    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6f9t\" (UniqueName: \"kubernetes.io/projected/a4ee9138-0121-4b07-a24f-2b86c4ea10af-kube-api-access-q6f9t\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:16.689399 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.689261    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4ee9138-0121-4b07-a24f-2b86c4ea10af-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:16.892281 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.892246    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"]
Sep 11 17:58:16.898009 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:16.897971    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9aab53-cdfdd6b79-zfc4p"]
Sep 11 17:58:17.782155 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:17.782114    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ee9138-0121-4b07-a24f-2b86c4ea10af" path="/var/lib/kubelet/pods/a4ee9138-0121-4b07-a24f-2b86c4ea10af/volumes"
Sep 11 17:58:18.523267 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:18.523239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 17:58:19.802354 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.802329    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:58:19.924324 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924294    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-certs\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924324 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-socket\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-envoy\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-data\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istiod-ca-cert\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22bxv\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-kube-api-access-22bxv\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924569    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-podinfo\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-credential-socket\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924901 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:19.924901 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924699    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-token\") pod \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\" (UID: \"f32a26e1-0037-4122-9b7c-e5b3164ef34d\") "
Sep 11 17:58:19.924901 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:19.924901 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.924795    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:19.925184 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.925057    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:19.925184 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.925077    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:19.925184 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.925088    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:19.925338 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.925277    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:19.925338 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.925288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:58:19.927546 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.927518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-kube-api-access-22bxv" (OuterVolumeSpecName: "kube-api-access-22bxv") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "kube-api-access-22bxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:19.927546 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.927519    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:58:19.927696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.927549    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:58:19.927696 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:19.927564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f32a26e1-0037-4122-9b7c-e5b3164ef34d" (UID: "f32a26e1-0037-4122-9b7c-e5b3164ef34d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:58:20.025692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.025655    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:20.025692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.025688    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:20.025692 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.025698    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:20.025916 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.025707    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bxv\" (UniqueName: \"kubernetes.io/projected/f32a26e1-0037-4122-9b7c-e5b3164ef34d-kube-api-access-22bxv\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:20.025916 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.025716    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f32a26e1-0037-4122-9b7c-e5b3164ef34d-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:20.025916 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.025724    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f32a26e1-0037-4122-9b7c-e5b3164ef34d-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 17:58:20.566357 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.566324    2563 generic.go:356] "Generic (PLEG): container finished" podID="f32a26e1-0037-4122-9b7c-e5b3164ef34d" containerID="0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e" exitCode=0
Sep 11 17:58:20.566567 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.566376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"
Sep 11 17:58:20.566567 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.566382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm" event={"ID":"f32a26e1-0037-4122-9b7c-e5b3164ef34d","Type":"ContainerDied","Data":"0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e"}
Sep 11 17:58:20.566567 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.566426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm" event={"ID":"f32a26e1-0037-4122-9b7c-e5b3164ef34d","Type":"ContainerDied","Data":"9474078f032ebf406f282a91dd768e513a6a6ae0f3fd27f1dbef08cab62ce5a4"}
Sep 11 17:58:20.566567 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.566447    2563 scope.go:117] "RemoveContainer" containerID="0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e"
Sep 11 17:58:20.577327 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.577309    2563 scope.go:117] "RemoveContainer" containerID="0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e"
Sep 11 17:58:20.577637 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:20.577618    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e\": container with ID starting with 0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e not found: ID does not exist" containerID="0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e"
Sep 11 17:58:20.577722 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.577651    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e"} err="failed to get container status \"0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e\": rpc error: code = NotFound desc = could not find container \"0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e\": container with ID starting with 0d1bddef219b9e501602551d4e6310302f872345b6660e25587b60de7931686e not found: ID does not exist"
Sep 11 17:58:20.592156 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.592124    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"]
Sep 11 17:58:20.599130 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:20.599101    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-6baa6b-openshift-default-7758d4b648-skfnm"]
Sep 11 17:58:21.781999 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:21.781960    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f32a26e1-0037-4122-9b7c-e5b3164ef34d" path="/var/lib/kubelet/pods/f32a26e1-0037-4122-9b7c-e5b3164ef34d/volumes"
Sep 11 17:58:22.511431 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:22.511399    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 17:58:22.511645 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:22.511524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 17:58:25.367529 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.367475    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"]
Sep 11 17:58:25.368978 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.368936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ee9138-0121-4b07-a24f-2b86c4ea10af" containerName="maas-api"
Sep 11 17:58:25.369184 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.369170    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ee9138-0121-4b07-a24f-2b86c4ea10af" containerName="maas-api"
Sep 11 17:58:25.369328 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.369316    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32a26e1-0037-4122-9b7c-e5b3164ef34d" containerName="istio-proxy"
Sep 11 17:58:25.369431 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.369421    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32a26e1-0037-4122-9b7c-e5b3164ef34d" containerName="istio-proxy"
Sep 11 17:58:25.369768 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.369754    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32a26e1-0037-4122-9b7c-e5b3164ef34d" containerName="istio-proxy"
Sep 11 17:58:25.369907 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.369881    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ee9138-0121-4b07-a24f-2b86c4ea10af" containerName="maas-api"
Sep 11 17:58:25.375024 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.375002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.378470 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.378445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a18ac6-openshift-default-dockercfg-m8qvv\""
Sep 11 17:58:25.388076 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.388048    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"]
Sep 11 17:58:25.480919 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.480877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtjz\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-kube-api-access-9rtjz\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481244 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-data\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481424 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-podinfo\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481628 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-envoy\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-certs\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481770 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-socket\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istiod-ca-cert\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-token\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.481964 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.481940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-credential-socket\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.582608 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-envoy\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.582775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-certs\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.582775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-socket\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.582775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istiod-ca-cert\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.582775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-token\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.582775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-credential-socket\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtjz\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-kube-api-access-9rtjz\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-data\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583015 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.582861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-podinfo\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583207 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.583145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-socket\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583269 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.583232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-credential-socket\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583383 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.583350    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-certs\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.583411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istiod-ca-cert\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.583708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.583504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-data\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.585122 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.585099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-envoy\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.585345 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.585328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-podinfo\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.594258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.594234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-token\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.594458 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.594437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtjz\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-kube-api-access-9rtjz\") pod \"e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") " pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.686851 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.686770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:25.972284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.972084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"]
Sep 11 17:58:25.975279 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:58:25.975249    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97493b51_0ea8_4ada_9caa_8b6cf3a2815d.slice/crio-fc8d9318e18f9547e2ab477074f83a533b1122a75fd10fe4673bd29cd8d760de WatchSource:0}: Error finding container fc8d9318e18f9547e2ab477074f83a533b1122a75fd10fe4673bd29cd8d760de: Status 404 returned error can't find the container with id fc8d9318e18f9547e2ab477074f83a533b1122a75fd10fe4673bd29cd8d760de
Sep 11 17:58:25.978131 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.978091    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:58:25.978204 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.978183    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:58:25.978252 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:25.978237    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:58:26.612954 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:26.612918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" event={"ID":"97493b51-0ea8-4ada-9caa-8b6cf3a2815d","Type":"ContainerStarted","Data":"19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"}
Sep 11 17:58:26.613362 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:26.612975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" event={"ID":"97493b51-0ea8-4ada-9caa-8b6cf3a2815d","Type":"ContainerStarted","Data":"fc8d9318e18f9547e2ab477074f83a533b1122a75fd10fe4673bd29cd8d760de"}
Sep 11 17:58:26.636986 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:26.636932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podStartSLOduration=1.6369126760000001 podStartE2EDuration="1.636912676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:26.634537898 +0000 UTC m=+1173.397992625" watchObservedRunningTime="2026-09-11 17:58:26.636912676 +0000 UTC m=+1173.400367404"
Sep 11 17:58:26.687258 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:26.687218    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:27.693863 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:27.693832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:28.627441 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:28.627403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:28.628541 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:28.628519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:58:33.141288 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.141249    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"]
Sep 11 17:58:33.145702 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.145674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.171028 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.170994    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"]
Sep 11 17:58:33.174805 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.174774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a18ac6-serving-cert\""
Sep 11 17:58:33.264708 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.264674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.264908 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.264744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvkl\" (UniqueName: \"kubernetes.io/projected/cae5fbd5-471a-4424-b922-9dcc423b072e-kube-api-access-hkvkl\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.264908 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.264839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.264908 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.264892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae5fbd5-471a-4424-b922-9dcc423b072e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.366116 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.366074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.366300 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.366130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae5fbd5-471a-4424-b922-9dcc423b072e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.366300 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.366219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.366300 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:33.366231    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-serving-cert: secret "maas-api-e2e-shared-a-w3-a18ac6-serving-cert" not found
Sep 11 17:58:33.366300 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.366258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvkl\" (UniqueName: \"kubernetes.io/projected/cae5fbd5-471a-4424-b922-9dcc423b072e-kube-api-access-hkvkl\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.366466 ip-10-0-135-167 kubenswrapper[2563]: E0911 17:58:33.366312    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls podName:cae5fbd5-471a-4424-b922-9dcc423b072e nodeName:}" failed. No retries permitted until 2026-09-11 17:58:33.866295858 +0000 UTC m=+1180.629750568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" (UID: "cae5fbd5-471a-4424-b922-9dcc423b072e") : secret "maas-api-e2e-shared-a-w3-a18ac6-serving-cert" not found
Sep 11 17:58:33.366836 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.366810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae5fbd5-471a-4424-b922-9dcc423b072e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.368840 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.368823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.393159 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.393084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvkl\" (UniqueName: \"kubernetes.io/projected/cae5fbd5-471a-4424-b922-9dcc423b072e-kube-api-access-hkvkl\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.873238 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.873204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:33.875775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:33.875751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:34.059935 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:34.059776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:34.348292 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:34.348266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"]
Sep 11 17:58:34.348941 ip-10-0-135-167 kubenswrapper[2563]: W0911 17:58:34.348914    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae5fbd5_471a_4424_b922_9dcc423b072e.slice/crio-b226405ae7a29dcb3d2063ae68d3722d904d1e63a698928f8398badb771f9147 WatchSource:0}: Error finding container b226405ae7a29dcb3d2063ae68d3722d904d1e63a698928f8398badb771f9147: Status 404 returned error can't find the container with id b226405ae7a29dcb3d2063ae68d3722d904d1e63a698928f8398badb771f9147
Sep 11 17:58:34.669006 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:34.668904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" event={"ID":"cae5fbd5-471a-4424-b922-9dcc423b072e","Type":"ContainerStarted","Data":"b226405ae7a29dcb3d2063ae68d3722d904d1e63a698928f8398badb771f9147"}
Sep 11 17:58:35.676849 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:35.676811    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" event={"ID":"cae5fbd5-471a-4424-b922-9dcc423b072e","Type":"ContainerStarted","Data":"f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64"}
Sep 11 17:58:35.677284 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:35.676889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:35.722168 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:35.722105    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" podStartSLOduration=2.238485259 podStartE2EDuration="2.722083299s" totalImagesPullingTime="483.59804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:35.721707852 +0000 UTC m=+1182.485162581" watchObservedRunningTime="2026-09-11 17:58:35.722083299 +0000 UTC m=+1182.485538029"
Sep 11 17:58:41.687745 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:41.687714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 17:58:53.844762 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:53.844722    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:58:53.848479 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:58:53.848451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 17:59:02.872086 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:02.872046    2563 generic.go:356] "Generic (PLEG): container finished" podID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerID="19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596" exitCode=0
Sep 11 17:59:02.872557 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:02.872122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" event={"ID":"97493b51-0ea8-4ada-9caa-8b6cf3a2815d","Type":"ContainerDied","Data":"19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"}
Sep 11 17:59:02.872609 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:02.872570    2563 scope.go:117] "RemoveContainer" containerID="19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"
Sep 11 17:59:02.873781 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:02.873747    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:59:02.873908 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:02.873853    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:59:02.873908 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:02.873902    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 17:59:03.881627 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:03.881581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" event={"ID":"97493b51-0ea8-4ada-9caa-8b6cf3a2815d","Type":"ContainerStarted","Data":"ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9"}
Sep 11 17:59:04.687811 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:04.687760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:59:04.688092 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:04.688062    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 11 17:59:04.688190 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:04.688129    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 11 17:59:05.687648 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:05.687603    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 11 17:59:05.688147 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:05.687673    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 11 17:59:06.690727 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:06.690682    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 11 17:59:06.691192 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:06.690750    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 11 17:59:07.687828 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:07.687787    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 11 17:59:07.688169 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:07.688134    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 11 17:59:08.053680 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.053586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"]
Sep 11 17:59:08.057839 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.057810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.060939 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.060913    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1bf937-serving-cert\""
Sep 11 17:59:08.070076 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.070052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"]
Sep 11 17:59:08.161305 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.161268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ab737-7caa-426b-a271-8650aa3a69b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.161523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.161358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvc7\" (UniqueName: \"kubernetes.io/projected/9c7ab737-7caa-426b-a271-8650aa3a69b8-kube-api-access-bzvc7\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.161523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.161379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.161523 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.161422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.262581 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.262540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.262775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.262630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ab737-7caa-426b-a271-8650aa3a69b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.262775 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.262747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvc7\" (UniqueName: \"kubernetes.io/projected/9c7ab737-7caa-426b-a271-8650aa3a69b8-kube-api-access-bzvc7\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.262892 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.262774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.263396 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.263372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ab737-7caa-426b-a271-8650aa3a69b8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.265150 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.265127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.265339 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.265320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.270981 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.270953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvc7\" (UniqueName: \"kubernetes.io/projected/9c7ab737-7caa-426b-a271-8650aa3a69b8-kube-api-access-bzvc7\") pod \"maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.370693 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.370659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:08.692074 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.691045    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 11 17:59:08.692074 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.691099    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 11 17:59:08.702061 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.701868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"]
Sep 11 17:59:08.927607 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:08.927483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v" event={"ID":"9c7ab737-7caa-426b-a271-8650aa3a69b8","Type":"ContainerStarted","Data":"52597a590a1c2c379822b681a8769f339e9e606d9d6e3b9c89ccba8713088ae7"}
Sep 11 17:59:09.692835 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:09.692751    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:59:09.693256 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:09.693048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:59:09.693736 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:09.693720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 17:59:09.935626 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:09.935591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v" event={"ID":"9c7ab737-7caa-426b-a271-8650aa3a69b8","Type":"ContainerStarted","Data":"ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe"}
Sep 11 17:59:09.935823 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:09.935669    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 17:59:09.956241 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:09.956131    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v" podStartSLOduration=1.296446312 podStartE2EDuration="1.95611361s" totalImagesPullingTime="659.667298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:09.954384719 +0000 UTC m=+1216.717839446" watchObservedRunningTime="2026-09-11 17:59:09.95611361 +0000 UTC m=+1216.719568337"
Sep 11 17:59:15.947102 ip-10-0-135-167 kubenswrapper[2563]: I0911 17:59:15.947071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 18:00:49.299392 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.299356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"]
Sep 11 18:00:49.300184 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.300148    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" podUID="b3c5d68b-f884-4487-8a6c-65bfa1564016" containerName="maas-api" containerID="cri-o://883ee53438dca120fd642966d10afc6a82a1379888cedd50e078f0acf4c51f46" gracePeriod=30
Sep 11 18:00:49.384426 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.384393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"]
Sep 11 18:00:49.384691 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.384662    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn" podUID="895dcca2-8198-43f0-b57b-f013ddf83189" containerName="payload-processing" containerID="cri-o://4c40ea766e501796cf1228f9c17fb9d279d60aa798bbb74cb6576fbbdfcd6329" gracePeriod=30
Sep 11 18:00:49.394884 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.394853    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"]
Sep 11 18:00:49.395728 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.395096    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw" podUID="09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" containerName="payload-pre-processing" containerID="cri-o://7ad338c323e5375af3e5b5c214310bc396603c4a449bda7420d14dad1b3a5e02" gracePeriod=30
Sep 11 18:00:49.576375 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.575145    2563 generic.go:356] "Generic (PLEG): container finished" podID="895dcca2-8198-43f0-b57b-f013ddf83189" containerID="4c40ea766e501796cf1228f9c17fb9d279d60aa798bbb74cb6576fbbdfcd6329" exitCode=0
Sep 11 18:00:49.576375 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.575301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn" event={"ID":"895dcca2-8198-43f0-b57b-f013ddf83189","Type":"ContainerDied","Data":"4c40ea766e501796cf1228f9c17fb9d279d60aa798bbb74cb6576fbbdfcd6329"}
Sep 11 18:00:49.578208 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.577833    2563 generic.go:356] "Generic (PLEG): container finished" podID="09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" containerID="7ad338c323e5375af3e5b5c214310bc396603c4a449bda7420d14dad1b3a5e02" exitCode=0
Sep 11 18:00:49.578208 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.577923    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw" event={"ID":"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187","Type":"ContainerDied","Data":"7ad338c323e5375af3e5b5c214310bc396603c4a449bda7420d14dad1b3a5e02"}
Sep 11 18:00:49.580984 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.580695    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3c5d68b-f884-4487-8a6c-65bfa1564016" containerID="883ee53438dca120fd642966d10afc6a82a1379888cedd50e078f0acf4c51f46" exitCode=0
Sep 11 18:00:49.580984 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.580869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" event={"ID":"b3c5d68b-f884-4487-8a6c-65bfa1564016","Type":"ContainerDied","Data":"883ee53438dca120fd642966d10afc6a82a1379888cedd50e078f0acf4c51f46"}
Sep 11 18:00:49.672688 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.672618    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 18:00:49.735551 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.735519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 18:00:49.760468 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.760440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 18:00:49.776931 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.776900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c5d68b-f884-4487-8a6c-65bfa1564016-openshift-service-ca\") pod \"b3c5d68b-f884-4487-8a6c-65bfa1564016\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") "
Sep 11 18:00:49.777145 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.777079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dkbb\" (UniqueName: \"kubernetes.io/projected/b3c5d68b-f884-4487-8a6c-65bfa1564016-kube-api-access-9dkbb\") pod \"b3c5d68b-f884-4487-8a6c-65bfa1564016\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") "
Sep 11 18:00:49.777145 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.777122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-metrics-certs\") pod \"b3c5d68b-f884-4487-8a6c-65bfa1564016\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") "
Sep 11 18:00:49.777273 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.777163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls\") pod \"b3c5d68b-f884-4487-8a6c-65bfa1564016\" (UID: \"b3c5d68b-f884-4487-8a6c-65bfa1564016\") "
Sep 11 18:00:49.777391 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.777352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c5d68b-f884-4487-8a6c-65bfa1564016-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3c5d68b-f884-4487-8a6c-65bfa1564016" (UID: "b3c5d68b-f884-4487-8a6c-65bfa1564016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:49.777554 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.777525    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c5d68b-f884-4487-8a6c-65bfa1564016-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.782390 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.781757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3c5d68b-f884-4487-8a6c-65bfa1564016" (UID: "b3c5d68b-f884-4487-8a6c-65bfa1564016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:49.782721 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.782579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c5d68b-f884-4487-8a6c-65bfa1564016-kube-api-access-9dkbb" (OuterVolumeSpecName: "kube-api-access-9dkbb") pod "b3c5d68b-f884-4487-8a6c-65bfa1564016" (UID: "b3c5d68b-f884-4487-8a6c-65bfa1564016"). InnerVolumeSpecName "kube-api-access-9dkbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:49.783316 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.783280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3c5d68b-f884-4487-8a6c-65bfa1564016" (UID: "b3c5d68b-f884-4487-8a6c-65bfa1564016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:49.878354 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878264    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddffs\" (UniqueName: \"kubernetes.io/projected/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-kube-api-access-ddffs\") pod \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") "
Sep 11 18:00:49.878354 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895dcca2-8198-43f0-b57b-f013ddf83189-plugins-config-volume\") pod \"895dcca2-8198-43f0-b57b-f013ddf83189\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") "
Sep 11 18:00:49.878621 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878382    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-plugins-config-volume\") pod \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\" (UID: \"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187\") "
Sep 11 18:00:49.878621 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwd2r\" (UniqueName: \"kubernetes.io/projected/895dcca2-8198-43f0-b57b-f013ddf83189-kube-api-access-pwd2r\") pod \"895dcca2-8198-43f0-b57b-f013ddf83189\" (UID: \"895dcca2-8198-43f0-b57b-f013ddf83189\") "
Sep 11 18:00:49.878778 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878755    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkbb\" (UniqueName: \"kubernetes.io/projected/b3c5d68b-f884-4487-8a6c-65bfa1564016-kube-api-access-9dkbb\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.878847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878785    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.878847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878801    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c5d68b-f884-4487-8a6c-65bfa1564016-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.878847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" (UID: "09ff55e3-5b4c-42f0-b5cd-564ccc7ff187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:49.878970 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.878837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895dcca2-8198-43f0-b57b-f013ddf83189-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "895dcca2-8198-43f0-b57b-f013ddf83189" (UID: "895dcca2-8198-43f0-b57b-f013ddf83189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:49.880790 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.880763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895dcca2-8198-43f0-b57b-f013ddf83189-kube-api-access-pwd2r" (OuterVolumeSpecName: "kube-api-access-pwd2r") pod "895dcca2-8198-43f0-b57b-f013ddf83189" (UID: "895dcca2-8198-43f0-b57b-f013ddf83189"). InnerVolumeSpecName "kube-api-access-pwd2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:49.880985 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.880966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-kube-api-access-ddffs" (OuterVolumeSpecName: "kube-api-access-ddffs") pod "09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" (UID: "09ff55e3-5b4c-42f0-b5cd-564ccc7ff187"). InnerVolumeSpecName "kube-api-access-ddffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:49.979814 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.979778    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddffs\" (UniqueName: \"kubernetes.io/projected/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-kube-api-access-ddffs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.979814 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.979813    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/895dcca2-8198-43f0-b57b-f013ddf83189-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.979814 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.979823    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:49.980056 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:49.979844    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwd2r\" (UniqueName: \"kubernetes.io/projected/895dcca2-8198-43f0-b57b-f013ddf83189-kube-api-access-pwd2r\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:00:50.589056 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.589025    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"
Sep 11 18:00:50.589579 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.589061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn" event={"ID":"895dcca2-8198-43f0-b57b-f013ddf83189","Type":"ContainerDied","Data":"5c6efbb1c5ac02e1a2b030206979ab3211d7426fcfbfcfef28a05587eced9ac9"}
Sep 11 18:00:50.589579 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.589132    2563 scope.go:117] "RemoveContainer" containerID="4c40ea766e501796cf1228f9c17fb9d279d60aa798bbb74cb6576fbbdfcd6329"
Sep 11 18:00:50.590767 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.590728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw" event={"ID":"09ff55e3-5b4c-42f0-b5cd-564ccc7ff187","Type":"ContainerDied","Data":"ef916ac97b5dd4d778188b4af5eb555b4b93560a111cf442533aac3877b3ca1e"}
Sep 11 18:00:50.590850 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.590741    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"
Sep 11 18:00:50.592184 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.592144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw" event={"ID":"b3c5d68b-f884-4487-8a6c-65bfa1564016","Type":"ContainerDied","Data":"a2d77fad31bcc0fa6a26afb046f942f6884c2dd82dc4ef3e7c0e8bd5cf1efded"}
Sep 11 18:00:50.592184 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.592176    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"
Sep 11 18:00:50.607603 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.607574    2563 scope.go:117] "RemoveContainer" containerID="7ad338c323e5375af3e5b5c214310bc396603c4a449bda7420d14dad1b3a5e02"
Sep 11 18:00:50.618320 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.618291    2563 scope.go:117] "RemoveContainer" containerID="883ee53438dca120fd642966d10afc6a82a1379888cedd50e078f0acf4c51f46"
Sep 11 18:00:50.622613 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.622586    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"]
Sep 11 18:00:50.631312 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.631281    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ed18394d-689c96cf7-zrljn"]
Sep 11 18:00:50.644908 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.644875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"]
Sep 11 18:00:50.651262 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.651236    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ed18394d-5d9f9c9c99-sqlnw"]
Sep 11 18:00:50.665318 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.665287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"]
Sep 11 18:00:50.667969 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:50.667947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ed18394d-7f986fc7db-tgrgw"]
Sep 11 18:00:51.782263 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:51.782231    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" path="/var/lib/kubelet/pods/09ff55e3-5b4c-42f0-b5cd-564ccc7ff187/volumes"
Sep 11 18:00:51.782659 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:51.782630    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895dcca2-8198-43f0-b57b-f013ddf83189" path="/var/lib/kubelet/pods/895dcca2-8198-43f0-b57b-f013ddf83189/volumes"
Sep 11 18:00:51.782984 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:51.782970    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c5d68b-f884-4487-8a6c-65bfa1564016" path="/var/lib/kubelet/pods/b3c5d68b-f884-4487-8a6c-65bfa1564016/volumes"
Sep 11 18:00:59.181315 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181280    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"]
Sep 11 18:00:59.181737 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181724    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c5d68b-f884-4487-8a6c-65bfa1564016" containerName="maas-api"
Sep 11 18:00:59.181788 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181739    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c5d68b-f884-4487-8a6c-65bfa1564016" containerName="maas-api"
Sep 11 18:00:59.181788 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181749    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895dcca2-8198-43f0-b57b-f013ddf83189" containerName="payload-processing"
Sep 11 18:00:59.181788 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181755    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="895dcca2-8198-43f0-b57b-f013ddf83189" containerName="payload-processing"
Sep 11 18:00:59.181788 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181773    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" containerName="payload-pre-processing"
Sep 11 18:00:59.181788 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181779    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" containerName="payload-pre-processing"
Sep 11 18:00:59.181937 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181847    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="895dcca2-8198-43f0-b57b-f013ddf83189" containerName="payload-processing"
Sep 11 18:00:59.181937 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181858    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c5d68b-f884-4487-8a6c-65bfa1564016" containerName="maas-api"
Sep 11 18:00:59.181937 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.181871    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ff55e3-5b4c-42f0-b5cd-564ccc7ff187" containerName="payload-pre-processing"
Sep 11 18:00:59.187798 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.187773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.191113 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.191082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1bf937\"/\"kube-root-ca.crt\""
Sep 11 18:00:59.191113 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.191105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1bf937\"/\"auth-test-model-1bf937-kserve-self-signed-certs\""
Sep 11 18:00:59.191331 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.191195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1bf937\"/\"default-dockercfg-8p6ch\""
Sep 11 18:00:59.191457 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.191441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1bf937\"/\"openshift-service-ca.crt\""
Sep 11 18:00:59.198562 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.198538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"]
Sep 11 18:00:59.266602 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.266566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-model-cache\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.266783 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.266614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tls-certs\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.266783 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.266707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4h4p\" (UniqueName: \"kubernetes.io/projected/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-kube-api-access-s4h4p\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.266783 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.266761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-home\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.266902 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.266788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-dshm\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.266902 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.266847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tmp-dir\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.367903 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.367860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-model-cache\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.367903 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.367904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tls-certs\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368161 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.367958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4h4p\" (UniqueName: \"kubernetes.io/projected/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-kube-api-access-s4h4p\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368161 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.367993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-home\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368161 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.368027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-dshm\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368161 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.368084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tmp-dir\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368387 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.368327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-model-cache\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368437 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.368388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-home\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.368472 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.368458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tmp-dir\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.370521 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.370478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-dshm\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.370689 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.370673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tls-certs\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.377200 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.377177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4h4p\" (UniqueName: \"kubernetes.io/projected/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-kube-api-access-s4h4p\") pod \"auth-test-model-1bf937-kserve-b89dc84cf-x26f4\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") " pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.498311 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.498203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:00:59.776351 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:00:59.776321    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b7f3b7c_87bb_45fd_9844_ffdab51ace39.slice/crio-36550c76dc8e238ef340a3a7ef4b4e35d9bc2fb712b02afcebf6368ecb97ce38 WatchSource:0}: Error finding container 36550c76dc8e238ef340a3a7ef4b4e35d9bc2fb712b02afcebf6368ecb97ce38: Status 404 returned error can't find the container with id 36550c76dc8e238ef340a3a7ef4b4e35d9bc2fb712b02afcebf6368ecb97ce38
Sep 11 18:00:59.783349 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:00:59.783324    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"]
Sep 11 18:01:00.663247 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:00.663199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" event={"ID":"2b7f3b7c-87bb-45fd-9844-ffdab51ace39","Type":"ContainerStarted","Data":"a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399"}
Sep 11 18:01:00.663844 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:00.663821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" event={"ID":"2b7f3b7c-87bb-45fd-9844-ffdab51ace39","Type":"ContainerStarted","Data":"36550c76dc8e238ef340a3a7ef4b4e35d9bc2fb712b02afcebf6368ecb97ce38"}
Sep 11 18:01:00.685435 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:00.685367    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" podStartSLOduration=1.685344217 podStartE2EDuration="1.685344217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:01:00.684304242 +0000 UTC m=+1327.447758969" watchObservedRunningTime="2026-09-11 18:01:00.685344217 +0000 UTC m=+1327.448798945"
Sep 11 18:01:09.498856 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:09.498813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:01:09.499274 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:09.498877    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:01:09.511510 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:09.511453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:01:09.726024 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:09.725991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:01:41.885727 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.885692    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"]
Sep 11 18:01:41.889558 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.889531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:41.892504 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.892466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ab253\"/\"kube-root-ca.crt\""
Sep 11 18:01:41.893921 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.893896    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ab253\"/\"openshift-service-ca.crt\""
Sep 11 18:01:41.893921 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.893918    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ab253\"/\"distinct-2-w4-4ab253-kserve-self-signed-certs\""
Sep 11 18:01:41.894141 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.894017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ab253\"/\"default-dockercfg-8n74d\""
Sep 11 18:01:41.901087 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:41.901063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"]
Sep 11 18:01:42.017480 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.017440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-model-cache\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.017480 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.017482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tmp-dir\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.017727 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.017557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhdzz\" (UniqueName: \"kubernetes.io/projected/f6e8a32a-ea99-4c92-9c9e-6342892fad96-kube-api-access-lhdzz\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.017727 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.017584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tls-certs\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.017727 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.017667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-home\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.017727 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.017718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-dshm\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119048 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-home\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119247 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-dshm\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119247 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-model-cache\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119247 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tmp-dir\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119247 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhdzz\" (UniqueName: \"kubernetes.io/projected/f6e8a32a-ea99-4c92-9c9e-6342892fad96-kube-api-access-lhdzz\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119247 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tls-certs\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119541 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-home\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119613 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-model-cache\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.119665 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.119607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tmp-dir\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.121642 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.121621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-dshm\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.121887 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.121869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tls-certs\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.128784 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.128756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhdzz\" (UniqueName: \"kubernetes.io/projected/f6e8a32a-ea99-4c92-9c9e-6342892fad96-kube-api-access-lhdzz\") pod \"distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") " pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.201333 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.201238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:42.490076 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.489984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"]
Sep 11 18:01:42.494147 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:01:42.494109    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6e8a32a_ea99_4c92_9c9e_6342892fad96.slice/crio-8c42aabc822b5c03dd8b6dc07670c868be58903264a2e1f66e8c100217914151 WatchSource:0}: Error finding container 8c42aabc822b5c03dd8b6dc07670c868be58903264a2e1f66e8c100217914151: Status 404 returned error can't find the container with id 8c42aabc822b5c03dd8b6dc07670c868be58903264a2e1f66e8c100217914151
Sep 11 18:01:42.918387 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.918352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5" event={"ID":"f6e8a32a-ea99-4c92-9c9e-6342892fad96","Type":"ContainerStarted","Data":"7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d"}
Sep 11 18:01:42.918387 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.918394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5" event={"ID":"f6e8a32a-ea99-4c92-9c9e-6342892fad96","Type":"ContainerStarted","Data":"8c42aabc822b5c03dd8b6dc07670c868be58903264a2e1f66e8c100217914151"}
Sep 11 18:01:42.935828 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:42.935774    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5" podStartSLOduration=1.93575319 podStartE2EDuration="1.93575319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:01:42.934693978 +0000 UTC m=+1369.698148707" watchObservedRunningTime="2026-09-11 18:01:42.93575319 +0000 UTC m=+1369.699207918"
Sep 11 18:01:52.201693 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:52.201649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:52.202120 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:52.201712    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:52.214409 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:52.214383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:01:52.994719 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:01:52.994688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:02:49.740381 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.740300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"]
Sep 11 18:02:49.744919 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.744893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.748100 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.748078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-95f9c9-serving-cert\""
Sep 11 18:02:49.754095 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.754066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"]
Sep 11 18:02:49.840170 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.840135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-metrics-certs\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.840170 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.840174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w26b\" (UniqueName: \"kubernetes.io/projected/56869b51-0fd3-4e42-9a59-aa6013e42774-kube-api-access-7w26b\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.840385 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.840195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56869b51-0fd3-4e42-9a59-aa6013e42774-openshift-service-ca\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.840429 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.840375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.942031 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.941993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.942223 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.942057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-metrics-certs\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.942223 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.942084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w26b\" (UniqueName: \"kubernetes.io/projected/56869b51-0fd3-4e42-9a59-aa6013e42774-kube-api-access-7w26b\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.942223 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.942100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56869b51-0fd3-4e42-9a59-aa6013e42774-openshift-service-ca\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.942223 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:02:49.942140    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-serving-cert: secret "maas-api-e2e-api-a-95f9c9-serving-cert" not found
Sep 11 18:02:49.942426 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:02:49.942236    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls podName:56869b51-0fd3-4e42-9a59-aa6013e42774 nodeName:}" failed. No retries permitted until 2026-09-11 18:02:50.442217893 +0000 UTC m=+1437.205672598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls") pod "maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" (UID: "56869b51-0fd3-4e42-9a59-aa6013e42774") : secret "maas-api-e2e-api-a-95f9c9-serving-cert" not found
Sep 11 18:02:49.942864 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.942847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56869b51-0fd3-4e42-9a59-aa6013e42774-openshift-service-ca\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.944761 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.944736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-metrics-certs\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:49.953803 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:49.953783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w26b\" (UniqueName: \"kubernetes.io/projected/56869b51-0fd3-4e42-9a59-aa6013e42774-kube-api-access-7w26b\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:50.352197 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.352161    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"]
Sep 11 18:02:50.356111 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.356089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.361173 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.361143    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-95f9c9\""
Sep 11 18:02:50.380643 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.380610    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"]
Sep 11 18:02:50.400684 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.400639    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"]
Sep 11 18:02:50.404327 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.404306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.410347 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.410323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-95f9c9-dockercfg-bcddh\""
Sep 11 18:02:50.439614 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.439574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"]
Sep 11 18:02:50.447348 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.447314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:50.447522 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.447391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpvf\" (UniqueName: \"kubernetes.io/projected/cdde9284-4e06-4857-a2dd-cf0cbc07306e-kube-api-access-qvpvf\") pod \"payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.447582 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.447515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdde9284-4e06-4857-a2dd-cf0cbc07306e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.450369 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.450329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls\") pod \"maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:50.548853 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.548813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpvf\" (UniqueName: \"kubernetes.io/projected/cdde9284-4e06-4857-a2dd-cf0cbc07306e-kube-api-access-qvpvf\") pod \"payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.549043 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.548890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdde9284-4e06-4857-a2dd-cf0cbc07306e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.549043 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.548930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswc4\" (UniqueName: \"kubernetes.io/projected/d9fcd755-e213-4a35-a006-d9ba7c7ed987-kube-api-access-cswc4\") pod \"payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") " pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.549043 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.548988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcd755-e213-4a35-a006-d9ba7c7ed987-plugins-config-volume\") pod \"payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") " pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.549633 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.549612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdde9284-4e06-4857-a2dd-cf0cbc07306e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.557717 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.557683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpvf\" (UniqueName: \"kubernetes.io/projected/cdde9284-4e06-4857-a2dd-cf0cbc07306e-kube-api-access-qvpvf\") pod \"payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.650640 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.650534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswc4\" (UniqueName: \"kubernetes.io/projected/d9fcd755-e213-4a35-a006-d9ba7c7ed987-kube-api-access-cswc4\") pod \"payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") " pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.650640 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.650611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcd755-e213-4a35-a006-d9ba7c7ed987-plugins-config-volume\") pod \"payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") " pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.651283 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.651261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcd755-e213-4a35-a006-d9ba7c7ed987-plugins-config-volume\") pod \"payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") " pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.659320 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.659289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswc4\" (UniqueName: \"kubernetes.io/projected/d9fcd755-e213-4a35-a006-d9ba7c7ed987-kube-api-access-cswc4\") pod \"payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") " pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.659460 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.659445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:50.666625 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.666604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:50.715572 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.715347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:50.941245 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.941213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"]
Sep 11 18:02:50.942875 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:02:50.942850    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56869b51_0fd3_4e42_9a59_aa6013e42774.slice/crio-ff67be8e79af8df5dba67b8c94a658b2fb6765177af3e1037709f7e5ce2d16c9 WatchSource:0}: Error finding container ff67be8e79af8df5dba67b8c94a658b2fb6765177af3e1037709f7e5ce2d16c9: Status 404 returned error can't find the container with id ff67be8e79af8df5dba67b8c94a658b2fb6765177af3e1037709f7e5ce2d16c9
Sep 11 18:02:50.944341 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.944318    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:02:50.954872 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:50.954667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"]
Sep 11 18:02:50.957129 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:02:50.957102    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde9284_4e06_4857_a2dd_cf0cbc07306e.slice/crio-e4a494f0a863c73f0b780fcb92878146ee8510ba392554c2687ac89dbca7fdc6 WatchSource:0}: Error finding container e4a494f0a863c73f0b780fcb92878146ee8510ba392554c2687ac89dbca7fdc6: Status 404 returned error can't find the container with id e4a494f0a863c73f0b780fcb92878146ee8510ba392554c2687ac89dbca7fdc6
Sep 11 18:02:51.008264 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.008240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"]
Sep 11 18:02:51.010257 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:02:51.010228    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fcd755_e213_4a35_a006_d9ba7c7ed987.slice/crio-922367b13bffdcf87cfea0701cd94822212870bd9911beebd7940543588cff82 WatchSource:0}: Error finding container 922367b13bffdcf87cfea0701cd94822212870bd9911beebd7940543588cff82: Status 404 returned error can't find the container with id 922367b13bffdcf87cfea0701cd94822212870bd9911beebd7940543588cff82
Sep 11 18:02:51.343029 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.342993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n" event={"ID":"cdde9284-4e06-4857-a2dd-cf0cbc07306e","Type":"ContainerStarted","Data":"8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b"}
Sep 11 18:02:51.343244 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.343054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n" event={"ID":"cdde9284-4e06-4857-a2dd-cf0cbc07306e","Type":"ContainerStarted","Data":"e4a494f0a863c73f0b780fcb92878146ee8510ba392554c2687ac89dbca7fdc6"}
Sep 11 18:02:51.343244 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.343204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:02:51.344403 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.344376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" event={"ID":"56869b51-0fd3-4e42-9a59-aa6013e42774","Type":"ContainerStarted","Data":"ff67be8e79af8df5dba67b8c94a658b2fb6765177af3e1037709f7e5ce2d16c9"}
Sep 11 18:02:51.346447 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.346418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k" event={"ID":"d9fcd755-e213-4a35-a006-d9ba7c7ed987","Type":"ContainerStarted","Data":"437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9"}
Sep 11 18:02:51.346599 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.346482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k" event={"ID":"d9fcd755-e213-4a35-a006-d9ba7c7ed987","Type":"ContainerStarted","Data":"922367b13bffdcf87cfea0701cd94822212870bd9911beebd7940543588cff82"}
Sep 11 18:02:51.346599 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.346549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:02:51.360890 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.360842    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n" podStartSLOduration=1.360826549 podStartE2EDuration="1.360826549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:51.360577663 +0000 UTC m=+1438.124032389" watchObservedRunningTime="2026-09-11 18:02:51.360826549 +0000 UTC m=+1438.124281275"
Sep 11 18:02:51.383041 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:51.382989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k" podStartSLOduration=1.382970208 podStartE2EDuration="1.382970208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:51.380228186 +0000 UTC m=+1438.143682915" watchObservedRunningTime="2026-09-11 18:02:51.382970208 +0000 UTC m=+1438.146424935"
Sep 11 18:02:52.354716 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:52.354676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" event={"ID":"56869b51-0fd3-4e42-9a59-aa6013e42774","Type":"ContainerStarted","Data":"def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d"}
Sep 11 18:02:52.355179 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:52.354956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:02:52.375832 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:52.375773    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" podStartSLOduration=2.549482634 podStartE2EDuration="3.375755031s" totalImagesPullingTime="826.272397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:52.374153035 +0000 UTC m=+1439.137607761" watchObservedRunningTime="2026-09-11 18:02:52.375755031 +0000 UTC m=+1439.139209772"
Sep 11 18:02:58.366724 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:02:58.366694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:03:02.357194 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:02.357161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:03:02.357631 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:02.357237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:03:23.510616 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.510562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"]
Sep 11 18:03:23.511035 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.510843    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879" podUID="f60cfefc-2c06-4d92-9e4e-42aff11ef803" containerName="maas-api" containerID="cri-o://96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a" gracePeriod=30
Sep 11 18:03:23.781730 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.781700    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 18:03:23.921578 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.920963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60cfefc-2c06-4d92-9e4e-42aff11ef803-openshift-service-ca\") pod \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") "
Sep 11 18:03:23.921578 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.921046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59bph\" (UniqueName: \"kubernetes.io/projected/f60cfefc-2c06-4d92-9e4e-42aff11ef803-kube-api-access-59bph\") pod \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") "
Sep 11 18:03:23.921578 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.921114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-metrics-certs\") pod \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") "
Sep 11 18:03:23.921578 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.921180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-maas-api-tls\") pod \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\" (UID: \"f60cfefc-2c06-4d92-9e4e-42aff11ef803\") "
Sep 11 18:03:23.921578 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.921564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60cfefc-2c06-4d92-9e4e-42aff11ef803-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f60cfefc-2c06-4d92-9e4e-42aff11ef803" (UID: "f60cfefc-2c06-4d92-9e4e-42aff11ef803"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:03:23.927369 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.927332    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60cfefc-2c06-4d92-9e4e-42aff11ef803-kube-api-access-59bph" (OuterVolumeSpecName: "kube-api-access-59bph") pod "f60cfefc-2c06-4d92-9e4e-42aff11ef803" (UID: "f60cfefc-2c06-4d92-9e4e-42aff11ef803"). InnerVolumeSpecName "kube-api-access-59bph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:23.930590 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.930553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f60cfefc-2c06-4d92-9e4e-42aff11ef803" (UID: "f60cfefc-2c06-4d92-9e4e-42aff11ef803"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:23.932796 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:23.932740    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f60cfefc-2c06-4d92-9e4e-42aff11ef803" (UID: "f60cfefc-2c06-4d92-9e4e-42aff11ef803"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:24.022634 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.022542    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:24.022634 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.022575    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60cfefc-2c06-4d92-9e4e-42aff11ef803-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:24.022634 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.022585    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59bph\" (UniqueName: \"kubernetes.io/projected/f60cfefc-2c06-4d92-9e4e-42aff11ef803-kube-api-access-59bph\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:24.022634 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.022594    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60cfefc-2c06-4d92-9e4e-42aff11ef803-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:24.565347 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.565312    2563 generic.go:356] "Generic (PLEG): container finished" podID="f60cfefc-2c06-4d92-9e4e-42aff11ef803" containerID="96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a" exitCode=0
Sep 11 18:03:24.565769 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.565357    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"
Sep 11 18:03:24.565769 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.565401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879" event={"ID":"f60cfefc-2c06-4d92-9e4e-42aff11ef803","Type":"ContainerDied","Data":"96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a"}
Sep 11 18:03:24.565769 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.565457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879" event={"ID":"f60cfefc-2c06-4d92-9e4e-42aff11ef803","Type":"ContainerDied","Data":"2f416da83add4d0d314d49946ea3bcccd0cd59fd5708d597225cb1eb36ddb846"}
Sep 11 18:03:24.565769 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.565480    2563 scope.go:117] "RemoveContainer" containerID="96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a"
Sep 11 18:03:24.576312 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.576292    2563 scope.go:117] "RemoveContainer" containerID="96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a"
Sep 11 18:03:24.576639 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:03:24.576619    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a\": container with ID starting with 96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a not found: ID does not exist" containerID="96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a"
Sep 11 18:03:24.576689 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.576648    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a"} err="failed to get container status \"96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a\": rpc error: code = NotFound desc = could not find container \"96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a\": container with ID starting with 96aff81fa21ce9430a5226fbad3f5938a625c56d7da3a6a142d965151585684a not found: ID does not exist"
Sep 11 18:03:24.593519 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.593461    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"]
Sep 11 18:03:24.596465 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:24.596440    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-286c27-84c79b57b-8m879"]
Sep 11 18:03:25.782612 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:25.782576    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60cfefc-2c06-4d92-9e4e-42aff11ef803" path="/var/lib/kubelet/pods/f60cfefc-2c06-4d92-9e4e-42aff11ef803/volumes"
Sep 11 18:03:29.149685 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.149655    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"]
Sep 11 18:03:29.150126 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.150113    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60cfefc-2c06-4d92-9e4e-42aff11ef803" containerName="maas-api"
Sep 11 18:03:29.150126 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.150128    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60cfefc-2c06-4d92-9e4e-42aff11ef803" containerName="maas-api"
Sep 11 18:03:29.150248 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.150205    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60cfefc-2c06-4d92-9e4e-42aff11ef803" containerName="maas-api"
Sep 11 18:03:29.154966 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.154937    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.157766 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.157742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a93550-serving-cert\""
Sep 11 18:03:29.164817 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.164794    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"]
Sep 11 18:03:29.280096 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.280061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-maas-api-tls\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.280276 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.280183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-openshift-service-ca\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.280276 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.280230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndn9\" (UniqueName: \"kubernetes.io/projected/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-kube-api-access-rndn9\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.280276 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.280260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-metrics-certs\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.381961 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.381920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndn9\" (UniqueName: \"kubernetes.io/projected/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-kube-api-access-rndn9\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.382143 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.381967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-metrics-certs\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.382143 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.382043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-maas-api-tls\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.382143 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.382120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-openshift-service-ca\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.382812 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.382786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-openshift-service-ca\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.384679 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.384660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-metrics-certs\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.384783 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.384766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-maas-api-tls\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.391394 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.391371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndn9\" (UniqueName: \"kubernetes.io/projected/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-kube-api-access-rndn9\") pod \"maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.468650 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.468554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:29.756690 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:29.756656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"]
Sep 11 18:03:29.757575 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:03:29.757550    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc41c9c_abd1_46a0_a4e5_b3211ad3bae7.slice/crio-ebab71f0dc83b4569119b276aa91c851f70b5d5741e331a82d429d41e68eb769 WatchSource:0}: Error finding container ebab71f0dc83b4569119b276aa91c851f70b5d5741e331a82d429d41e68eb769: Status 404 returned error can't find the container with id ebab71f0dc83b4569119b276aa91c851f70b5d5741e331a82d429d41e68eb769
Sep 11 18:03:30.612200 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:30.612166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg" event={"ID":"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7","Type":"ContainerStarted","Data":"8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac"}
Sep 11 18:03:30.612620 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:30.612217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg" event={"ID":"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7","Type":"ContainerStarted","Data":"ebab71f0dc83b4569119b276aa91c851f70b5d5741e331a82d429d41e68eb769"}
Sep 11 18:03:30.612620 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:30.612286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:31.453219 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:31.453144    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg" podStartSLOduration=1.971015354 podStartE2EDuration="2.453118045s" totalImagesPullingTime="482.102691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:30.634076329 +0000 UTC m=+1477.397531056" watchObservedRunningTime="2026-09-11 18:03:31.453118045 +0000 UTC m=+1478.216572773"
Sep 11 18:03:31.454818 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:31.454787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"]
Sep 11 18:03:31.455145 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:31.455116    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy" containerID="cri-o://783ca1e75c45cc7d0202950fb22e14d0b57bd6d0738b50648d3a14563f6e9904" gracePeriod=30
Sep 11 18:03:36.624068 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.624033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:03:36.661690 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.661655    2563 generic.go:356] "Generic (PLEG): container finished" podID="97a9a098-d486-4015-9253-107d0a3ab937" containerID="783ca1e75c45cc7d0202950fb22e14d0b57bd6d0738b50648d3a14563f6e9904" exitCode=0
Sep 11 18:03:36.661872 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.661834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" event={"ID":"97a9a098-d486-4015-9253-107d0a3ab937","Type":"ContainerDied","Data":"783ca1e75c45cc7d0202950fb22e14d0b57bd6d0738b50648d3a14563f6e9904"}
Sep 11 18:03:36.661961 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.661897    2563 scope.go:117] "RemoveContainer" containerID="0a0b077d808911a146c4b12b1b20b739fdfd8932575b89dbd72c332e4f3b857e"
Sep 11 18:03:36.730781 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.730710    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 18:03:36.757835 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.757805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a9a098-d486-4015-9253-107d0a3ab937-istiod-ca-cert\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758036 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.757844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-data\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758036 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.757867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-istio-token\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758036 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.757901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-certs\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758036 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.757971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-credential-socket\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758036 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w7dw\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-kube-api-access-4w7dw\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758307 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-envoy\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758307 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a9a098-d486-4015-9253-107d0a3ab937-istio-podinfo\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758307 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-socket\") pod \"97a9a098-d486-4015-9253-107d0a3ab937\" (UID: \"97a9a098-d486-4015-9253-107d0a3ab937\") "
Sep 11 18:03:36.758307 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-data" (OuterVolumeSpecName: "istio-data") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:36.758562 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:36.758623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a9a098-d486-4015-9253-107d0a3ab937-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:03:36.758730 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:36.758730 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758715    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.758730 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:36.759147 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.758737    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.760648 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.760621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:03:36.760804 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.760785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-istio-token" (OuterVolumeSpecName: "istio-token") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:36.761074 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.761046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a9a098-d486-4015-9253-107d0a3ab937-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:03:36.761159 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.761092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-kube-api-access-4w7dw" (OuterVolumeSpecName: "kube-api-access-4w7dw") pod "97a9a098-d486-4015-9253-107d0a3ab937" (UID: "97a9a098-d486-4015-9253-107d0a3ab937"). InnerVolumeSpecName "kube-api-access-4w7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:36.859314 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859270    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a9a098-d486-4015-9253-107d0a3ab937-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.859314 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859314    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.859314 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859327    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.859649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859342    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w7dw\" (UniqueName: \"kubernetes.io/projected/97a9a098-d486-4015-9253-107d0a3ab937-kube-api-access-4w7dw\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.859649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859355    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.859649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859465    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a9a098-d486-4015-9253-107d0a3ab937-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:36.859649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:36.859524    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a9a098-d486-4015-9253-107d0a3ab937-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:03:37.669216 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:37.669186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"
Sep 11 18:03:37.669650 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:37.669233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh" event={"ID":"97a9a098-d486-4015-9253-107d0a3ab937","Type":"ContainerDied","Data":"eedf746a1276b986f19d61414bc88f9a5595f3bce5a820a28963e641b5bd36a2"}
Sep 11 18:03:37.669650 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:37.669291    2563 scope.go:117] "RemoveContainer" containerID="783ca1e75c45cc7d0202950fb22e14d0b57bd6d0738b50648d3a14563f6e9904"
Sep 11 18:03:37.692602 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:37.692543    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"]
Sep 11 18:03:37.696038 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:37.696004    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-286c27-openshift-default-6c59b96fdd-dkhrh"]
Sep 11 18:03:37.782816 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:37.782781    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a9a098-d486-4015-9253-107d0a3ab937" path="/var/lib/kubelet/pods/97a9a098-d486-4015-9253-107d0a3ab937/volumes"
Sep 11 18:03:53.895640 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:53.895605    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:03:53.898941 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:03:53.898914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:04:12.776270 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:12.776227    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"]
Sep 11 18:04:12.778958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:12.776504    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" podUID="56869b51-0fd3-4e42-9a59-aa6013e42774" containerName="maas-api" containerID="cri-o://def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d" gracePeriod=30
Sep 11 18:04:12.836318 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:12.836280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"]
Sep 11 18:04:12.836569 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:12.836542    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k" podUID="d9fcd755-e213-4a35-a006-d9ba7c7ed987" containerName="payload-processing" containerID="cri-o://437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9" gracePeriod=30
Sep 11 18:04:12.847900 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:12.847871    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"]
Sep 11 18:04:12.848147 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:12.848123    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n" podUID="cdde9284-4e06-4857-a2dd-cf0cbc07306e" containerName="payload-pre-processing" containerID="cri-o://8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b" gracePeriod=30
Sep 11 18:04:13.191828 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.191801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:04:13.194899 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.194872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:04:13.198621 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.198603    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:04:13.250685 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcd755-e213-4a35-a006-d9ba7c7ed987-plugins-config-volume\") pod \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") "
Sep 11 18:04:13.250881 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvpvf\" (UniqueName: \"kubernetes.io/projected/cdde9284-4e06-4857-a2dd-cf0cbc07306e-kube-api-access-qvpvf\") pod \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") "
Sep 11 18:04:13.250881 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w26b\" (UniqueName: \"kubernetes.io/projected/56869b51-0fd3-4e42-9a59-aa6013e42774-kube-api-access-7w26b\") pod \"56869b51-0fd3-4e42-9a59-aa6013e42774\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") "
Sep 11 18:04:13.250881 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdde9284-4e06-4857-a2dd-cf0cbc07306e-plugins-config-volume\") pod \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\" (UID: \"cdde9284-4e06-4857-a2dd-cf0cbc07306e\") "
Sep 11 18:04:13.250881 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls\") pod \"56869b51-0fd3-4e42-9a59-aa6013e42774\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") "
Sep 11 18:04:13.251119 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-metrics-certs\") pod \"56869b51-0fd3-4e42-9a59-aa6013e42774\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") "
Sep 11 18:04:13.251119 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cswc4\" (UniqueName: \"kubernetes.io/projected/d9fcd755-e213-4a35-a006-d9ba7c7ed987-kube-api-access-cswc4\") pod \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\" (UID: \"d9fcd755-e213-4a35-a006-d9ba7c7ed987\") "
Sep 11 18:04:13.251119 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.250998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56869b51-0fd3-4e42-9a59-aa6013e42774-openshift-service-ca\") pod \"56869b51-0fd3-4e42-9a59-aa6013e42774\" (UID: \"56869b51-0fd3-4e42-9a59-aa6013e42774\") "
Sep 11 18:04:13.251119 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.251104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fcd755-e213-4a35-a006-d9ba7c7ed987-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9fcd755-e213-4a35-a006-d9ba7c7ed987" (UID: "d9fcd755-e213-4a35-a006-d9ba7c7ed987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:04:13.251732 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.251407    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56869b51-0fd3-4e42-9a59-aa6013e42774-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "56869b51-0fd3-4e42-9a59-aa6013e42774" (UID: "56869b51-0fd3-4e42-9a59-aa6013e42774"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:04:13.251732 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.251577    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9fcd755-e213-4a35-a006-d9ba7c7ed987-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.251732 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.251596    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56869b51-0fd3-4e42-9a59-aa6013e42774-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.251732 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.251691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdde9284-4e06-4857-a2dd-cf0cbc07306e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdde9284-4e06-4857-a2dd-cf0cbc07306e" (UID: "cdde9284-4e06-4857-a2dd-cf0cbc07306e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:04:13.254118 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.254071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "56869b51-0fd3-4e42-9a59-aa6013e42774" (UID: "56869b51-0fd3-4e42-9a59-aa6013e42774"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:13.254240 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.254070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdde9284-4e06-4857-a2dd-cf0cbc07306e-kube-api-access-qvpvf" (OuterVolumeSpecName: "kube-api-access-qvpvf") pod "cdde9284-4e06-4857-a2dd-cf0cbc07306e" (UID: "cdde9284-4e06-4857-a2dd-cf0cbc07306e"). InnerVolumeSpecName "kube-api-access-qvpvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:13.254240 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.254107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56869b51-0fd3-4e42-9a59-aa6013e42774-kube-api-access-7w26b" (OuterVolumeSpecName: "kube-api-access-7w26b") pod "56869b51-0fd3-4e42-9a59-aa6013e42774" (UID: "56869b51-0fd3-4e42-9a59-aa6013e42774"). InnerVolumeSpecName "kube-api-access-7w26b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:13.254466 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.254437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "56869b51-0fd3-4e42-9a59-aa6013e42774" (UID: "56869b51-0fd3-4e42-9a59-aa6013e42774"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:13.255828 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.255809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fcd755-e213-4a35-a006-d9ba7c7ed987-kube-api-access-cswc4" (OuterVolumeSpecName: "kube-api-access-cswc4") pod "d9fcd755-e213-4a35-a006-d9ba7c7ed987" (UID: "d9fcd755-e213-4a35-a006-d9ba7c7ed987"). InnerVolumeSpecName "kube-api-access-cswc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:13.352623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.352531    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswc4\" (UniqueName: \"kubernetes.io/projected/d9fcd755-e213-4a35-a006-d9ba7c7ed987-kube-api-access-cswc4\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.352623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.352562    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvpvf\" (UniqueName: \"kubernetes.io/projected/cdde9284-4e06-4857-a2dd-cf0cbc07306e-kube-api-access-qvpvf\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.352623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.352571    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w26b\" (UniqueName: \"kubernetes.io/projected/56869b51-0fd3-4e42-9a59-aa6013e42774-kube-api-access-7w26b\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.352623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.352581    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdde9284-4e06-4857-a2dd-cf0cbc07306e-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.352623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.352591    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.352623 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.352599    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56869b51-0fd3-4e42-9a59-aa6013e42774-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:13.897210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.897179    2563 generic.go:356] "Generic (PLEG): container finished" podID="56869b51-0fd3-4e42-9a59-aa6013e42774" containerID="def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d" exitCode=0
Sep 11 18:04:13.897683 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.897223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"
Sep 11 18:04:13.897683 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.897275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" event={"ID":"56869b51-0fd3-4e42-9a59-aa6013e42774","Type":"ContainerDied","Data":"def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d"}
Sep 11 18:04:13.897683 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.897326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6" event={"ID":"56869b51-0fd3-4e42-9a59-aa6013e42774","Type":"ContainerDied","Data":"ff67be8e79af8df5dba67b8c94a658b2fb6765177af3e1037709f7e5ce2d16c9"}
Sep 11 18:04:13.897683 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.897356    2563 scope.go:117] "RemoveContainer" containerID="def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d"
Sep 11 18:04:13.898949 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.898921    2563 generic.go:356] "Generic (PLEG): container finished" podID="d9fcd755-e213-4a35-a006-d9ba7c7ed987" containerID="437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9" exitCode=0
Sep 11 18:04:13.899070 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.898985    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"
Sep 11 18:04:13.899070 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.899001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k" event={"ID":"d9fcd755-e213-4a35-a006-d9ba7c7ed987","Type":"ContainerDied","Data":"437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9"}
Sep 11 18:04:13.899070 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.899051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k" event={"ID":"d9fcd755-e213-4a35-a006-d9ba7c7ed987","Type":"ContainerDied","Data":"922367b13bffdcf87cfea0701cd94822212870bd9911beebd7940543588cff82"}
Sep 11 18:04:13.901664 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.901033    2563 generic.go:356] "Generic (PLEG): container finished" podID="cdde9284-4e06-4857-a2dd-cf0cbc07306e" containerID="8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b" exitCode=0
Sep 11 18:04:13.901664 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.901078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n" event={"ID":"cdde9284-4e06-4857-a2dd-cf0cbc07306e","Type":"ContainerDied","Data":"8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b"}
Sep 11 18:04:13.901664 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.901115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n" event={"ID":"cdde9284-4e06-4857-a2dd-cf0cbc07306e","Type":"ContainerDied","Data":"e4a494f0a863c73f0b780fcb92878146ee8510ba392554c2687ac89dbca7fdc6"}
Sep 11 18:04:13.901664 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.901169    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"
Sep 11 18:04:13.910713 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.910695    2563 scope.go:117] "RemoveContainer" containerID="def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d"
Sep 11 18:04:13.911007 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:13.910987    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d\": container with ID starting with def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d not found: ID does not exist" containerID="def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d"
Sep 11 18:04:13.911083 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.911020    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d"} err="failed to get container status \"def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d\": rpc error: code = NotFound desc = could not find container \"def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d\": container with ID starting with def6eb0d3a8d02b1f244bab20fcc0d2b8b1ee2b2bd406a279740449501a50a3d not found: ID does not exist"
Sep 11 18:04:13.911083 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.911039    2563 scope.go:117] "RemoveContainer" containerID="437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9"
Sep 11 18:04:13.920164 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.920143    2563 scope.go:117] "RemoveContainer" containerID="437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9"
Sep 11 18:04:13.920511 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:13.920477    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9\": container with ID starting with 437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9 not found: ID does not exist" containerID="437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9"
Sep 11 18:04:13.920571 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.920518    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9"} err="failed to get container status \"437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9\": rpc error: code = NotFound desc = could not find container \"437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9\": container with ID starting with 437499c12f54e70d55eb5f4117c5e2fe398abfd803251f4112a14433e610bbe9 not found: ID does not exist"
Sep 11 18:04:13.920571 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.920536    2563 scope.go:117] "RemoveContainer" containerID="8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b"
Sep 11 18:04:13.925079 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.925053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"]
Sep 11 18:04:13.929647 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.929622    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-95f9c9-58c7db7cc7-9tn9k"]
Sep 11 18:04:13.931051 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.931033    2563 scope.go:117] "RemoveContainer" containerID="8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b"
Sep 11 18:04:13.931368 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:13.931351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b\": container with ID starting with 8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b not found: ID does not exist" containerID="8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b"
Sep 11 18:04:13.931444 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.931374    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b"} err="failed to get container status \"8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b\": rpc error: code = NotFound desc = could not find container \"8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b\": container with ID starting with 8a9899f4f4362e92ef17a08fe573d6913c60d934b2265bbdb124e97695b33c4b not found: ID does not exist"
Sep 11 18:04:13.942947 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.942918    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"]
Sep 11 18:04:13.947392 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.947368    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95f9c9-858c94c58c-cg2h6"]
Sep 11 18:04:13.958671 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.958626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"]
Sep 11 18:04:13.962848 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:13.962822    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95f9c9-5c798dff66-csv5n"]
Sep 11 18:04:15.788288 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:15.788169    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56869b51-0fd3-4e42-9a59-aa6013e42774" path="/var/lib/kubelet/pods/56869b51-0fd3-4e42-9a59-aa6013e42774/volumes"
Sep 11 18:04:15.788786 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:15.788674    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdde9284-4e06-4857-a2dd-cf0cbc07306e" path="/var/lib/kubelet/pods/cdde9284-4e06-4857-a2dd-cf0cbc07306e/volumes"
Sep 11 18:04:15.789015 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:15.788999    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9fcd755-e213-4a35-a006-d9ba7c7ed987" path="/var/lib/kubelet/pods/d9fcd755-e213-4a35-a006-d9ba7c7ed987/volumes"
Sep 11 18:04:36.215120 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.215080    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"]
Sep 11 18:04:36.215636 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.215359    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg" podUID="6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" containerName="maas-api" containerID="cri-o://8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac" gracePeriod=30
Sep 11 18:04:36.499126 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.499094    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:04:36.596639 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.596609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-openshift-service-ca\") pod \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") "
Sep 11 18:04:36.597439 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.596693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-maas-api-tls\") pod \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") "
Sep 11 18:04:36.597439 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.596743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rndn9\" (UniqueName: \"kubernetes.io/projected/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-kube-api-access-rndn9\") pod \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") "
Sep 11 18:04:36.597439 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.596768    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-metrics-certs\") pod \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\" (UID: \"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7\") "
Sep 11 18:04:36.597439 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.597035    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" (UID: "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:04:36.597439 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.597292    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:36.599886 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.599815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" (UID: "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:36.600235 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.599941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-kube-api-access-rndn9" (OuterVolumeSpecName: "kube-api-access-rndn9") pod "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" (UID: "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7"). InnerVolumeSpecName "kube-api-access-rndn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:36.600703 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.600672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" (UID: "6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:36.698112 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.698080    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rndn9\" (UniqueName: \"kubernetes.io/projected/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-kube-api-access-rndn9\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:36.698112 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.698109    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:36.698325 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:36.698122    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:37.052855 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.052816    2563 generic.go:356] "Generic (PLEG): container finished" podID="6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" containerID="8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac" exitCode=0
Sep 11 18:04:37.052855 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.052862    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"
Sep 11 18:04:37.053082 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.052862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg" event={"ID":"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7","Type":"ContainerDied","Data":"8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac"}
Sep 11 18:04:37.053082 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.052969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg" event={"ID":"6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7","Type":"ContainerDied","Data":"ebab71f0dc83b4569119b276aa91c851f70b5d5741e331a82d429d41e68eb769"}
Sep 11 18:04:37.053082 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.052996    2563 scope.go:117] "RemoveContainer" containerID="8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac"
Sep 11 18:04:37.063300 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.063280    2563 scope.go:117] "RemoveContainer" containerID="8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac"
Sep 11 18:04:37.063634 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:37.063613    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac\": container with ID starting with 8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac not found: ID does not exist" containerID="8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac"
Sep 11 18:04:37.063725 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.063643    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac"} err="failed to get container status \"8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac\": rpc error: code = NotFound desc = could not find container \"8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac\": container with ID starting with 8c3faff0b040f04c92093e312232d1fd0820e2e059e6285a25b6f79761936dac not found: ID does not exist"
Sep 11 18:04:37.077742 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.077706    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"]
Sep 11 18:04:37.080458 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.080435    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a93550-5c5458c7d6-c2ztg"]
Sep 11 18:04:37.783319 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:37.783287    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" path="/var/lib/kubelet/pods/6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7/volumes"
Sep 11 18:04:38.587248 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:38.587205    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"]
Sep 11 18:04:50.162077 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.162045    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"]
Sep 11 18:04:50.162525 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.162244    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" podUID="6845c5ec-5000-4613-8edc-d0d8d165027e" containerName="maas-api" containerID="cri-o://b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84" gracePeriod=30
Sep 11 18:04:50.423097 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.423028    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 18:04:50.549677 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.549644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-metrics-certs\") pod \"6845c5ec-5000-4613-8edc-d0d8d165027e\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") "
Sep 11 18:04:50.549875 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.549707    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7v2x\" (UniqueName: \"kubernetes.io/projected/6845c5ec-5000-4613-8edc-d0d8d165027e-kube-api-access-z7v2x\") pod \"6845c5ec-5000-4613-8edc-d0d8d165027e\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") "
Sep 11 18:04:50.549875 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.549731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6845c5ec-5000-4613-8edc-d0d8d165027e-openshift-service-ca\") pod \"6845c5ec-5000-4613-8edc-d0d8d165027e\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") "
Sep 11 18:04:50.549875 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.549805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls\") pod \"6845c5ec-5000-4613-8edc-d0d8d165027e\" (UID: \"6845c5ec-5000-4613-8edc-d0d8d165027e\") "
Sep 11 18:04:50.550140 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.550116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6845c5ec-5000-4613-8edc-d0d8d165027e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6845c5ec-5000-4613-8edc-d0d8d165027e" (UID: "6845c5ec-5000-4613-8edc-d0d8d165027e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:04:50.551868 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.551841    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6845c5ec-5000-4613-8edc-d0d8d165027e" (UID: "6845c5ec-5000-4613-8edc-d0d8d165027e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:50.551983 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.551897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6845c5ec-5000-4613-8edc-d0d8d165027e" (UID: "6845c5ec-5000-4613-8edc-d0d8d165027e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:50.552042 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.551988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6845c5ec-5000-4613-8edc-d0d8d165027e-kube-api-access-z7v2x" (OuterVolumeSpecName: "kube-api-access-z7v2x") pod "6845c5ec-5000-4613-8edc-d0d8d165027e" (UID: "6845c5ec-5000-4613-8edc-d0d8d165027e"). InnerVolumeSpecName "kube-api-access-z7v2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:50.651463 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.651424    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7v2x\" (UniqueName: \"kubernetes.io/projected/6845c5ec-5000-4613-8edc-d0d8d165027e-kube-api-access-z7v2x\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:50.651463 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.651456    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6845c5ec-5000-4613-8edc-d0d8d165027e-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:50.651463 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.651466    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:50.651463 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:50.651474    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6845c5ec-5000-4613-8edc-d0d8d165027e-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:51.138555 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.138516    2563 generic.go:356] "Generic (PLEG): container finished" podID="6845c5ec-5000-4613-8edc-d0d8d165027e" containerID="b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84" exitCode=0
Sep 11 18:04:51.138755 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.138581    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"
Sep 11 18:04:51.138755 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.138582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" event={"ID":"6845c5ec-5000-4613-8edc-d0d8d165027e","Type":"ContainerDied","Data":"b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84"}
Sep 11 18:04:51.138755 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.138665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf" event={"ID":"6845c5ec-5000-4613-8edc-d0d8d165027e","Type":"ContainerDied","Data":"6a2ce31b3385023f9445745bc7df66813efc4f33333fe0a2f924160f7cbde634"}
Sep 11 18:04:51.138755 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.138689    2563 scope.go:117] "RemoveContainer" containerID="b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84"
Sep 11 18:04:51.148919 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.148898    2563 scope.go:117] "RemoveContainer" containerID="b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84"
Sep 11 18:04:51.149259 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:51.149240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84\": container with ID starting with b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84 not found: ID does not exist" containerID="b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84"
Sep 11 18:04:51.149348 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.149266    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84"} err="failed to get container status \"b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84\": rpc error: code = NotFound desc = could not find container \"b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84\": container with ID starting with b5d69cbfb12c755273ab3284ceaedb8ee6e5b2c231128104838735f6a81fbd84 not found: ID does not exist"
Sep 11 18:04:51.164112 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.164079    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"]
Sep 11 18:04:51.166584 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.166557    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4ab253-574f6b7767-4fskf"]
Sep 11 18:04:51.783030 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:51.783000    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6845c5ec-5000-4613-8edc-d0d8d165027e" path="/var/lib/kubelet/pods/6845c5ec-5000-4613-8edc-d0d8d165027e/volumes"
Sep 11 18:04:53.607685 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.607640    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5" podUID="f6e8a32a-ea99-4c92-9c9e-6342892fad96" containerName="main" containerID="cri-o://7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d" gracePeriod=45
Sep 11 18:04:53.875182 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.875157    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:04:53.989261 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tmp-dir\") pod \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") "
Sep 11 18:04:53.989518 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989445    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhdzz\" (UniqueName: \"kubernetes.io/projected/f6e8a32a-ea99-4c92-9c9e-6342892fad96-kube-api-access-lhdzz\") pod \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") "
Sep 11 18:04:53.989624 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989532    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6e8a32a-ea99-4c92-9c9e-6342892fad96" (UID: "f6e8a32a-ea99-4c92-9c9e-6342892fad96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:04:53.989690 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tls-certs\") pod \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") "
Sep 11 18:04:53.989747 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-model-cache\") pod \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") "
Sep 11 18:04:53.989747 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-home\") pod \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") "
Sep 11 18:04:53.989846 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-dshm\") pod \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\" (UID: \"f6e8a32a-ea99-4c92-9c9e-6342892fad96\") "
Sep 11 18:04:53.990024 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.989975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6e8a32a-ea99-4c92-9c9e-6342892fad96" (UID: "f6e8a32a-ea99-4c92-9c9e-6342892fad96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:04:53.990138 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.990113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-home" (OuterVolumeSpecName: "home") pod "f6e8a32a-ea99-4c92-9c9e-6342892fad96" (UID: "f6e8a32a-ea99-4c92-9c9e-6342892fad96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:04:53.990315 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.990251    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tmp-dir\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:53.990450 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.990391    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-model-cache\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:53.990543 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.990454    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-home\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:53.992210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.992160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-dshm" (OuterVolumeSpecName: "dshm") pod "f6e8a32a-ea99-4c92-9c9e-6342892fad96" (UID: "f6e8a32a-ea99-4c92-9c9e-6342892fad96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:04:53.992210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.992173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6e8a32a-ea99-4c92-9c9e-6342892fad96" (UID: "f6e8a32a-ea99-4c92-9c9e-6342892fad96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:04:53.992363 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:53.992246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e8a32a-ea99-4c92-9c9e-6342892fad96-kube-api-access-lhdzz" (OuterVolumeSpecName: "kube-api-access-lhdzz") pod "f6e8a32a-ea99-4c92-9c9e-6342892fad96" (UID: "f6e8a32a-ea99-4c92-9c9e-6342892fad96"). InnerVolumeSpecName "kube-api-access-lhdzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:54.091719 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.091679    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhdzz\" (UniqueName: \"kubernetes.io/projected/f6e8a32a-ea99-4c92-9c9e-6342892fad96-kube-api-access-lhdzz\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:54.091719 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.091709    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6e8a32a-ea99-4c92-9c9e-6342892fad96-tls-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:54.091719 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.091719    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6e8a32a-ea99-4c92-9c9e-6342892fad96-dshm\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:54.159793 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.159696    2563 generic.go:356] "Generic (PLEG): container finished" podID="f6e8a32a-ea99-4c92-9c9e-6342892fad96" containerID="7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d" exitCode=0
Sep 11 18:04:54.159793 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.159745    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"
Sep 11 18:04:54.159793 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.159783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5" event={"ID":"f6e8a32a-ea99-4c92-9c9e-6342892fad96","Type":"ContainerDied","Data":"7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d"}
Sep 11 18:04:54.160068 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.159829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5" event={"ID":"f6e8a32a-ea99-4c92-9c9e-6342892fad96","Type":"ContainerDied","Data":"8c42aabc822b5c03dd8b6dc07670c868be58903264a2e1f66e8c100217914151"}
Sep 11 18:04:54.160068 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.159854    2563 scope.go:117] "RemoveContainer" containerID="7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d"
Sep 11 18:04:54.169737 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.169718    2563 scope.go:117] "RemoveContainer" containerID="7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d"
Sep 11 18:04:54.169998 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:54.169982    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d\": container with ID starting with 7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d not found: ID does not exist" containerID="7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d"
Sep 11 18:04:54.170048 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.170005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d"} err="failed to get container status \"7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d\": rpc error: code = NotFound desc = could not find container \"7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d\": container with ID starting with 7d106e743f864a5c2e714ddfedefe863c7433669cce5274be9a4e0bb5d3e1a1d not found: ID does not exist"
Sep 11 18:04:54.187233 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.187193    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"]
Sep 11 18:04:54.191428 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.191399    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ab253/distinct-2-w4-4ab253-kserve-768f5f5555-vvvr5"]
Sep 11 18:04:54.446579 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.446473    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"]
Sep 11 18:04:54.446783 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.446756    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4" podUID="c67faa52-d8cf-4a89-acff-89d3bc17a7f8" containerName="payload-pre-processing" containerID="cri-o://bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d" gracePeriod=30
Sep 11 18:04:54.733464 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.733431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 18:04:54.900681 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.900646    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-plugins-config-volume\") pod \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") "
Sep 11 18:04:54.900864 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.900704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkk7n\" (UniqueName: \"kubernetes.io/projected/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-kube-api-access-bkk7n\") pod \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\" (UID: \"c67faa52-d8cf-4a89-acff-89d3bc17a7f8\") "
Sep 11 18:04:54.901074 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.901050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c67faa52-d8cf-4a89-acff-89d3bc17a7f8" (UID: "c67faa52-d8cf-4a89-acff-89d3bc17a7f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:04:54.901158 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.901141    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:54.902932 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:54.902907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-kube-api-access-bkk7n" (OuterVolumeSpecName: "kube-api-access-bkk7n") pod "c67faa52-d8cf-4a89-acff-89d3bc17a7f8" (UID: "c67faa52-d8cf-4a89-acff-89d3bc17a7f8"). InnerVolumeSpecName "kube-api-access-bkk7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:04:55.002565 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.002455    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkk7n\" (UniqueName: \"kubernetes.io/projected/c67faa52-d8cf-4a89-acff-89d3bc17a7f8-kube-api-access-bkk7n\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:04:55.169834 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.169797    2563 generic.go:356] "Generic (PLEG): container finished" podID="c67faa52-d8cf-4a89-acff-89d3bc17a7f8" containerID="bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d" exitCode=0
Sep 11 18:04:55.170024 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.169862    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"
Sep 11 18:04:55.170024 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.169885    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4" event={"ID":"c67faa52-d8cf-4a89-acff-89d3bc17a7f8","Type":"ContainerDied","Data":"bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d"}
Sep 11 18:04:55.170024 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.169971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4" event={"ID":"c67faa52-d8cf-4a89-acff-89d3bc17a7f8","Type":"ContainerDied","Data":"40de19714e6a0ed828baab3251e02f1d6045cd74a7f1d2615869833db1178ad5"}
Sep 11 18:04:55.170024 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.170000    2563 scope.go:117] "RemoveContainer" containerID="bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d"
Sep 11 18:04:55.179563 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.179544    2563 scope.go:117] "RemoveContainer" containerID="bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d"
Sep 11 18:04:55.179863 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:04:55.179842    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d\": container with ID starting with bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d not found: ID does not exist" containerID="bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d"
Sep 11 18:04:55.179960 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.179872    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d"} err="failed to get container status \"bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d\": rpc error: code = NotFound desc = could not find container \"bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d\": container with ID starting with bfb86f6bf396636fb3e27328cc635c33c1e02b2567958ba641913d5ebee7320d not found: ID does not exist"
Sep 11 18:04:55.193503 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.193452    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"]
Sep 11 18:04:55.199825 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.199801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-21e472-744f8d5b66-zplv4"]
Sep 11 18:04:55.784586 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.784550    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c67faa52-d8cf-4a89-acff-89d3bc17a7f8" path="/var/lib/kubelet/pods/c67faa52-d8cf-4a89-acff-89d3bc17a7f8/volumes"
Sep 11 18:04:55.785001 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:04:55.784986    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e8a32a-ea99-4c92-9c9e-6342892fad96" path="/var/lib/kubelet/pods/f6e8a32a-ea99-4c92-9c9e-6342892fad96/volumes"
Sep 11 18:05:01.684137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:01.684097    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"]
Sep 11 18:05:01.684567 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:01.684418    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" containerID="cri-o://86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4" gracePeriod=30
Sep 11 18:05:03.061948 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:03.061909    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:05:03.062364 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:03.061979    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:05:06.931942 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:06.931912    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 18:05:07.123973 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.123928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-credential-socket\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.123973 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.123979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-data\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124218 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-certs\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124218 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8nxv\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-kube-api-access-l8nxv\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124218 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istiod-ca-cert\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124218 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-envoy\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124218 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124178    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-podinfo\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124218 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-token\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124589 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-socket\") pod \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\" (UID: \"e00c8a41-8b2c-46a3-a233-44a1a199aa79\") "
Sep 11 18:05:07.124589 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:07.124589 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-data" (OuterVolumeSpecName: "istio-data") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:07.124589 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:07.124779 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:05:07.124779 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:07.124779 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124678    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.124779 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124695    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.124779 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.124707    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.126517 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.126470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:07.126517 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.126479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:05:07.126903 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.126880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-token" (OuterVolumeSpecName: "istio-token") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:05:07.126903 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.126890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-kube-api-access-l8nxv" (OuterVolumeSpecName: "kube-api-access-l8nxv") pod "e00c8a41-8b2c-46a3-a233-44a1a199aa79" (UID: "e00c8a41-8b2c-46a3-a233-44a1a199aa79"). InnerVolumeSpecName "kube-api-access-l8nxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:05:07.226037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.225994    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8nxv\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-kube-api-access-l8nxv\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.226037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.226031    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.226037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.226044    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.226370 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.226051    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.226370 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.226059    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00c8a41-8b2c-46a3-a233-44a1a199aa79-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.226370 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.226067    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00c8a41-8b2c-46a3-a233-44a1a199aa79-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:07.244184 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.244154    2563 generic.go:356] "Generic (PLEG): container finished" podID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerID="86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4" exitCode=0
Sep 11 18:05:07.244339 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.244208    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"
Sep 11 18:05:07.244339 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.244250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" event={"ID":"e00c8a41-8b2c-46a3-a233-44a1a199aa79","Type":"ContainerDied","Data":"86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4"}
Sep 11 18:05:07.244339 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.244327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z" event={"ID":"e00c8a41-8b2c-46a3-a233-44a1a199aa79","Type":"ContainerDied","Data":"60a7b0f60e4ee77cdd6041124dad06d2f12050a85a6b868f0c0d531ff126c734"}
Sep 11 18:05:07.244468 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.244353    2563 scope.go:117] "RemoveContainer" containerID="86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4"
Sep 11 18:05:07.254264 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.254247    2563 scope.go:117] "RemoveContainer" containerID="d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"
Sep 11 18:05:07.263231 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.263212    2563 scope.go:117] "RemoveContainer" containerID="86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4"
Sep 11 18:05:07.263530 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:05:07.263510    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4\": container with ID starting with 86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4 not found: ID does not exist" containerID="86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4"
Sep 11 18:05:07.263585 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.263541    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4"} err="failed to get container status \"86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4\": rpc error: code = NotFound desc = could not find container \"86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4\": container with ID starting with 86dca6cfd8f5bdf88f98e10291add4d48c644839108a35a5fbd2deb7c2cc69d4 not found: ID does not exist"
Sep 11 18:05:07.263585 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.263561    2563 scope.go:117] "RemoveContainer" containerID="d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"
Sep 11 18:05:07.263832 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:05:07.263818    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6\": container with ID starting with d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6 not found: ID does not exist" containerID="d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"
Sep 11 18:05:07.263872 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.263836    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6"} err="failed to get container status \"d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6\": rpc error: code = NotFound desc = could not find container \"d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6\": container with ID starting with d4d9946d1413aecb119e8bd49bf4f9d20d93ca33ebfcb2ee31f1cfa404260ec6 not found: ID does not exist"
Sep 11 18:05:07.270899 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.270875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"]
Sep 11 18:05:07.278305 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.278280    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-21e472-openshift-default-bfc4cdb96-rr48z"]
Sep 11 18:05:07.782864 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:07.782825    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" path="/var/lib/kubelet/pods/e00c8a41-8b2c-46a3-a233-44a1a199aa79/volumes"
Sep 11 18:05:12.676805 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.676766    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"]
Sep 11 18:05:12.677302 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.677045    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" podUID="7fee485f-34a7-46a7-9a8e-8147773ae477" containerName="maas-api" containerID="cri-o://8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358" gracePeriod=30
Sep 11 18:05:12.934312 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.934246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 18:05:12.981087 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.981038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fee485f-34a7-46a7-9a8e-8147773ae477-openshift-service-ca\") pod \"7fee485f-34a7-46a7-9a8e-8147773ae477\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") "
Sep 11 18:05:12.981280 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.981101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls\") pod \"7fee485f-34a7-46a7-9a8e-8147773ae477\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") "
Sep 11 18:05:12.981280 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.981246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-665tt\" (UniqueName: \"kubernetes.io/projected/7fee485f-34a7-46a7-9a8e-8147773ae477-kube-api-access-665tt\") pod \"7fee485f-34a7-46a7-9a8e-8147773ae477\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") "
Sep 11 18:05:12.981280 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.981268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-metrics-certs\") pod \"7fee485f-34a7-46a7-9a8e-8147773ae477\" (UID: \"7fee485f-34a7-46a7-9a8e-8147773ae477\") "
Sep 11 18:05:12.981464 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.981390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fee485f-34a7-46a7-9a8e-8147773ae477-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fee485f-34a7-46a7-9a8e-8147773ae477" (UID: "7fee485f-34a7-46a7-9a8e-8147773ae477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:05:12.981660 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.981638    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fee485f-34a7-46a7-9a8e-8147773ae477-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:12.983522 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.983453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fee485f-34a7-46a7-9a8e-8147773ae477" (UID: "7fee485f-34a7-46a7-9a8e-8147773ae477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:05:12.983522 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.983480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fee485f-34a7-46a7-9a8e-8147773ae477" (UID: "7fee485f-34a7-46a7-9a8e-8147773ae477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:05:12.983668 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:12.983582    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fee485f-34a7-46a7-9a8e-8147773ae477-kube-api-access-665tt" (OuterVolumeSpecName: "kube-api-access-665tt") pod "7fee485f-34a7-46a7-9a8e-8147773ae477" (UID: "7fee485f-34a7-46a7-9a8e-8147773ae477"). InnerVolumeSpecName "kube-api-access-665tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:05:13.082933 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.082900    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:13.083140 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.083035    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665tt\" (UniqueName: \"kubernetes.io/projected/7fee485f-34a7-46a7-9a8e-8147773ae477-kube-api-access-665tt\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:13.083140 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.083075    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fee485f-34a7-46a7-9a8e-8147773ae477-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:13.285858 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.285759    2563 generic.go:356] "Generic (PLEG): container finished" podID="7fee485f-34a7-46a7-9a8e-8147773ae477" containerID="8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358" exitCode=0
Sep 11 18:05:13.285858 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.285821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"
Sep 11 18:05:13.285858 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.285832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" event={"ID":"7fee485f-34a7-46a7-9a8e-8147773ae477","Type":"ContainerDied","Data":"8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358"}
Sep 11 18:05:13.286126 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.285892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p" event={"ID":"7fee485f-34a7-46a7-9a8e-8147773ae477","Type":"ContainerDied","Data":"73c50e8217c155e58de4c07d6d106a0e13a7e675353b8efa00a3d197e4d242cf"}
Sep 11 18:05:13.286126 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.285912    2563 scope.go:117] "RemoveContainer" containerID="8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358"
Sep 11 18:05:13.298843 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.297728    2563 scope.go:117] "RemoveContainer" containerID="8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358"
Sep 11 18:05:13.298843 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:05:13.298784    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358\": container with ID starting with 8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358 not found: ID does not exist" containerID="8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358"
Sep 11 18:05:13.298843 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.298817    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358"} err="failed to get container status \"8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358\": rpc error: code = NotFound desc = could not find container \"8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358\": container with ID starting with 8085480073a29a52d803fa67052f2a3830bdbc917a1ddd26c882a5c934561358 not found: ID does not exist"
Sep 11 18:05:13.309791 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.309760    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"]
Sep 11 18:05:13.314346 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.314315    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7f85c3-6c97c44dbd-qxr8p"]
Sep 11 18:05:13.782201 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:13.782163    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fee485f-34a7-46a7-9a8e-8147773ae477" path="/var/lib/kubelet/pods/7fee485f-34a7-46a7-9a8e-8147773ae477/volumes"
Sep 11 18:05:24.602716 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:24.602675    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"]
Sep 11 18:05:39.622304 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:39.622240    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" podUID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" containerName="main" containerID="cri-o://a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399" gracePeriod=45
Sep 11 18:05:39.715803 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:39.715756    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" podUID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused"
Sep 11 18:05:39.878696 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:39.878634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:05:40.053469 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-dshm\") pod \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") "
Sep 11 18:05:40.053667 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4h4p\" (UniqueName: \"kubernetes.io/projected/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-kube-api-access-s4h4p\") pod \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") "
Sep 11 18:05:40.053667 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-model-cache\") pod \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") "
Sep 11 18:05:40.053743 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tls-certs\") pod \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") "
Sep 11 18:05:40.053804 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tmp-dir\") pod \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") "
Sep 11 18:05:40.053942 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-home\") pod \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\" (UID: \"2b7f3b7c-87bb-45fd-9844-ffdab51ace39\") "
Sep 11 18:05:40.053942 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.053866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b7f3b7c-87bb-45fd-9844-ffdab51ace39" (UID: "2b7f3b7c-87bb-45fd-9844-ffdab51ace39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:40.054116 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.054089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b7f3b7c-87bb-45fd-9844-ffdab51ace39" (UID: "2b7f3b7c-87bb-45fd-9844-ffdab51ace39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:40.054195 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.054122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-home" (OuterVolumeSpecName: "home") pod "2b7f3b7c-87bb-45fd-9844-ffdab51ace39" (UID: "2b7f3b7c-87bb-45fd-9844-ffdab51ace39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:40.054195 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.054169    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-model-cache\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:40.055912 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.055889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-kube-api-access-s4h4p" (OuterVolumeSpecName: "kube-api-access-s4h4p") pod "2b7f3b7c-87bb-45fd-9844-ffdab51ace39" (UID: "2b7f3b7c-87bb-45fd-9844-ffdab51ace39"). InnerVolumeSpecName "kube-api-access-s4h4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:05:40.055993 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.055974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b7f3b7c-87bb-45fd-9844-ffdab51ace39" (UID: "2b7f3b7c-87bb-45fd-9844-ffdab51ace39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:05:40.056032 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.056001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-dshm" (OuterVolumeSpecName: "dshm") pod "2b7f3b7c-87bb-45fd-9844-ffdab51ace39" (UID: "2b7f3b7c-87bb-45fd-9844-ffdab51ace39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:05:40.155831 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.155736    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tmp-dir\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:40.155831 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.155772    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-home\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:40.155831 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.155781    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-dshm\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:40.155831 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.155790    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4h4p\" (UniqueName: \"kubernetes.io/projected/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-kube-api-access-s4h4p\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:40.155831 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.155798    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7f3b7c-87bb-45fd-9844-ffdab51ace39-tls-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:05:40.441225 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.441133    2563 generic.go:356] "Generic (PLEG): container finished" podID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" containerID="a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399" exitCode=0
Sep 11 18:05:40.441225 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.441178    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"
Sep 11 18:05:40.441406 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.441225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" event={"ID":"2b7f3b7c-87bb-45fd-9844-ffdab51ace39","Type":"ContainerDied","Data":"a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399"}
Sep 11 18:05:40.441406 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.441267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4" event={"ID":"2b7f3b7c-87bb-45fd-9844-ffdab51ace39","Type":"ContainerDied","Data":"36550c76dc8e238ef340a3a7ef4b4e35d9bc2fb712b02afcebf6368ecb97ce38"}
Sep 11 18:05:40.441406 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.441288    2563 scope.go:117] "RemoveContainer" containerID="a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399"
Sep 11 18:05:40.451638 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.451622    2563 scope.go:117] "RemoveContainer" containerID="a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399"
Sep 11 18:05:40.451959 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:05:40.451941    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399\": container with ID starting with a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399 not found: ID does not exist" containerID="a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399"
Sep 11 18:05:40.452005 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.451969    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399"} err="failed to get container status \"a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399\": rpc error: code = NotFound desc = could not find container \"a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399\": container with ID starting with a22b28d8bb7edb720eb03f7965a111193ef4b66b5cd7fbb7214911b4a9185399 not found: ID does not exist"
Sep 11 18:05:40.469042 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.469010    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"]
Sep 11 18:05:40.475835 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:40.475808    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1bf937/auth-test-model-1bf937-kserve-b89dc84cf-x26f4"]
Sep 11 18:05:41.781927 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:05:41.781895    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" path="/var/lib/kubelet/pods/2b7f3b7c-87bb-45fd-9844-ffdab51ace39/volumes"
Sep 11 18:08:37.907574 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"]
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907943    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdde9284-4e06-4857-a2dd-cf0cbc07306e" containerName="payload-pre-processing"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907956    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdde9284-4e06-4857-a2dd-cf0cbc07306e" containerName="payload-pre-processing"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907964    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6845c5ec-5000-4613-8edc-d0d8d165027e" containerName="maas-api"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6845c5ec-5000-4613-8edc-d0d8d165027e" containerName="maas-api"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907978    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c67faa52-d8cf-4a89-acff-89d3bc17a7f8" containerName="payload-pre-processing"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907984    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c67faa52-d8cf-4a89-acff-89d3bc17a7f8" containerName="payload-pre-processing"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.907994    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56869b51-0fd3-4e42-9a59-aa6013e42774" containerName="maas-api"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="56869b51-0fd3-4e42-9a59-aa6013e42774" containerName="maas-api"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908010    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fee485f-34a7-46a7-9a8e-8147773ae477" containerName="maas-api"
Sep 11 18:08:37.908019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fee485f-34a7-46a7-9a8e-8147773ae477" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908028    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908033    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908049    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908062    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908068    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908074    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908081    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908093    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" containerName="main"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908101    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" containerName="main"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908107    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e8a32a-ea99-4c92-9c9e-6342892fad96" containerName="main"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908112    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e8a32a-ea99-4c92-9c9e-6342892fad96" containerName="main"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908118    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9fcd755-e213-4a35-a006-d9ba7c7ed987" containerName="payload-processing"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908123    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9fcd755-e213-4a35-a006-d9ba7c7ed987" containerName="payload-processing"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908188    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908194    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fee485f-34a7-46a7-9a8e-8147773ae477" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908201    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7f3b7c-87bb-45fd-9844-ffdab51ace39" containerName="main"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908208    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="56869b51-0fd3-4e42-9a59-aa6013e42774" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908214    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdde9284-4e06-4857-a2dd-cf0cbc07306e" containerName="payload-pre-processing"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908219    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6845c5ec-5000-4613-8edc-d0d8d165027e" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908225    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c67faa52-d8cf-4a89-acff-89d3bc17a7f8" containerName="payload-pre-processing"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908231    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908237    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908246    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e8a32a-ea99-4c92-9c9e-6342892fad96" containerName="main"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908254    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc41c9c-abd1-46a0-a4e5-b3211ad3bae7" containerName="maas-api"
Sep 11 18:08:37.908343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.908260    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9fcd755-e213-4a35-a006-d9ba7c7ed987" containerName="payload-processing"
Sep 11 18:08:37.911012 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.910996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.917998 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.917974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-64600a-openshift-default-dockercfg-pcdfv\""
Sep 11 18:08:37.934750 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.934716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"]
Sep 11 18:08:37.980913 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.980877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ea70a8-e42b-49d3-8672-179bdee19e2c-istiod-ca-cert\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981091 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.980957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-socket\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981091 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.980981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-data\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981091 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.981064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-token\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981091 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.981083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-credential-socket\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981294 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.981120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8db4\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-kube-api-access-s8db4\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981294 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.981145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-certs\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981294 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.981211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-envoy\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:37.981294 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:37.981242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-podinfo\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.081829 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8db4\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-kube-api-access-s8db4\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-certs\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-envoy\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-podinfo\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ea70a8-e42b-49d3-8672-179bdee19e2c-istiod-ca-cert\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-socket\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082037 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.081981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-data\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082379 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-token\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082379 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-credential-socket\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082531 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-certs\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082531 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-data\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082628 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-credential-socket\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082776 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-socket\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.082820 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.082805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ea70a8-e42b-49d3-8672-179bdee19e2c-istiod-ca-cert\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.084561 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.084535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-envoy\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.084665 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.084652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-podinfo\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.096738 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.096715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-token\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.097428 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.097404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8db4\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-kube-api-access-s8db4\") pod \"e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") " pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.221458 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.221356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:38.512760 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.512736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"]
Sep 11 18:08:38.514988 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:08:38.514967    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ea70a8_e42b_49d3_8672_179bdee19e2c.slice/crio-320708d6bdd13379e7fd8dd030ea05dc888721ab240b2c102b654499a114398d WatchSource:0}: Error finding container 320708d6bdd13379e7fd8dd030ea05dc888721ab240b2c102b654499a114398d: Status 404 returned error can't find the container with id 320708d6bdd13379e7fd8dd030ea05dc888721ab240b2c102b654499a114398d
Sep 11 18:08:38.517261 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.517227    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:08:38.517344 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.517295    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:08:38.517344 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:38.517330    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:08:39.442408 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:39.442373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" event={"ID":"85ea70a8-e42b-49d3-8672-179bdee19e2c","Type":"ContainerStarted","Data":"a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"}
Sep 11 18:08:39.442828 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:39.442424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" event={"ID":"85ea70a8-e42b-49d3-8672-179bdee19e2c","Type":"ContainerStarted","Data":"320708d6bdd13379e7fd8dd030ea05dc888721ab240b2c102b654499a114398d"}
Sep 11 18:08:39.470999 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:39.470944    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podStartSLOduration=2.470927457 podStartE2EDuration="2.470927457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:39.468781827 +0000 UTC m=+1786.232236566" watchObservedRunningTime="2026-09-11 18:08:39.470927457 +0000 UTC m=+1786.234382183"
Sep 11 18:08:40.222319 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:40.222279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:40.227413 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:40.227384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:40.448434 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:40.448398    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:40.449333 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:40.449314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:08:44.434689 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.434650    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"]
Sep 11 18:08:44.435076 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.435064    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy"
Sep 11 18:08:44.435076 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.435076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00c8a41-8b2c-46a3-a233-44a1a199aa79" containerName="istio-proxy"
Sep 11 18:08:44.435151 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.435139    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a9a098-d486-4015-9253-107d0a3ab937" containerName="istio-proxy"
Sep 11 18:08:44.438438 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.438417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.442157 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.442127    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-64600a\""
Sep 11 18:08:44.456594 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.456567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"]
Sep 11 18:08:44.514455 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.514373    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"]
Sep 11 18:08:44.517998 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.517975    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.524749 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.524667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-64600a-dockercfg-png99\""
Sep 11 18:08:44.552244 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.552203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqvc\" (UniqueName: \"kubernetes.io/projected/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-kube-api-access-qfqvc\") pod \"payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.552435 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.552256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.557445 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.557409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"]
Sep 11 18:08:44.653637 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.653604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhtn\" (UniqueName: \"kubernetes.io/projected/99d537b1-a65a-4966-89db-65073132255b-kube-api-access-5vhtn\") pod \"payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.653806 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.653712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d537b1-a65a-4966-89db-65073132255b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.653806 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.653748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqvc\" (UniqueName: \"kubernetes.io/projected/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-kube-api-access-qfqvc\") pod \"payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.653880 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.653856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.654538 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.654521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.666048 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.666021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqvc\" (UniqueName: \"kubernetes.io/projected/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-kube-api-access-qfqvc\") pod \"payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.750131 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.750041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:44.754449 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.754425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d537b1-a65a-4966-89db-65073132255b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.754591 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.754573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhtn\" (UniqueName: \"kubernetes.io/projected/99d537b1-a65a-4966-89db-65073132255b-kube-api-access-5vhtn\") pod \"payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.755139 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.755119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d537b1-a65a-4966-89db-65073132255b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.765302 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.765276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhtn\" (UniqueName: \"kubernetes.io/projected/99d537b1-a65a-4966-89db-65073132255b-kube-api-access-5vhtn\") pod \"payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:44.828974 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:44.828939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:45.039106 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.039074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"]
Sep 11 18:08:45.040953 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:08:45.040928    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dab35a1_6f01_4c5c_ace7_24cc82d8904d.slice/crio-26dc64457569fdc7e6c1485eaea82dd0ac58d5468f9ee2b99df60ea49798eca6 WatchSource:0}: Error finding container 26dc64457569fdc7e6c1485eaea82dd0ac58d5468f9ee2b99df60ea49798eca6: Status 404 returned error can't find the container with id 26dc64457569fdc7e6c1485eaea82dd0ac58d5468f9ee2b99df60ea49798eca6
Sep 11 18:08:45.126118 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.126093    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"]
Sep 11 18:08:45.128484 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:08:45.128453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d537b1_a65a_4966_89db_65073132255b.slice/crio-00034f924e9cff1194003c00fd24e7f9ca10d00cdb8b483b17cf916fd6ce61f2 WatchSource:0}: Error finding container 00034f924e9cff1194003c00fd24e7f9ca10d00cdb8b483b17cf916fd6ce61f2: Status 404 returned error can't find the container with id 00034f924e9cff1194003c00fd24e7f9ca10d00cdb8b483b17cf916fd6ce61f2
Sep 11 18:08:45.483863 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.483820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf" event={"ID":"99d537b1-a65a-4966-89db-65073132255b","Type":"ContainerStarted","Data":"707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9"}
Sep 11 18:08:45.484340 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.483890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf" event={"ID":"99d537b1-a65a-4966-89db-65073132255b","Type":"ContainerStarted","Data":"00034f924e9cff1194003c00fd24e7f9ca10d00cdb8b483b17cf916fd6ce61f2"}
Sep 11 18:08:45.484724 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.484698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:45.487206 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.487174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t" event={"ID":"4dab35a1-6f01-4c5c-ace7-24cc82d8904d","Type":"ContainerStarted","Data":"a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd"}
Sep 11 18:08:45.487327 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.487229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t" event={"ID":"4dab35a1-6f01-4c5c-ace7-24cc82d8904d","Type":"ContainerStarted","Data":"26dc64457569fdc7e6c1485eaea82dd0ac58d5468f9ee2b99df60ea49798eca6"}
Sep 11 18:08:45.487816 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.487793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:45.509191 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.509136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf" podStartSLOduration=1.509112103 podStartE2EDuration="1.509112103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:45.505962947 +0000 UTC m=+1792.269417675" watchObservedRunningTime="2026-09-11 18:08:45.509112103 +0000 UTC m=+1792.272566834"
Sep 11 18:08:45.529279 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:45.529222    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t" podStartSLOduration=1.52920413 podStartE2EDuration="1.52920413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:45.526145682 +0000 UTC m=+1792.289600410" watchObservedRunningTime="2026-09-11 18:08:45.52920413 +0000 UTC m=+1792.292658856"
Sep 11 18:08:53.938266 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:53.938237    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:08:53.939627 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:53.939598    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:08:57.502421 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:57.502387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:08:57.502820 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:57.502478    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:08:59.568264 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:59.568158    2563 generic.go:356] "Generic (PLEG): container finished" podID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerID="a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d" exitCode=0
Sep 11 18:08:59.568264 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:59.568233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" event={"ID":"85ea70a8-e42b-49d3-8672-179bdee19e2c","Type":"ContainerDied","Data":"a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"}
Sep 11 18:08:59.568730 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:59.568708    2563 scope.go:117] "RemoveContainer" containerID="a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"
Sep 11 18:08:59.569889 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:59.569857    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:08:59.570014 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:59.569948    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:08:59.570014 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:08:59.569992    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 11 18:09:00.597048 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:00.597010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" event={"ID":"85ea70a8-e42b-49d3-8672-179bdee19e2c","Type":"ContainerStarted","Data":"924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23"}
Sep 11 18:09:01.222104 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:01.222065    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:09:01.222337 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:01.222319    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 18:09:01.222397 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:01.222365    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 18:09:02.222651 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:02.222555    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 18:09:02.222651 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:02.222623    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 18:09:03.221991 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:03.221952    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 18:09:03.222189 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:03.222032    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 18:09:04.221878 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:04.221836    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 18:09:04.222280 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:04.221918    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 18:09:05.222230 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:05.222194    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 11 18:09:05.222661 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:05.222250    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 11 18:09:06.226670 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:06.226637    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:09:06.227143 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:06.227082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:09:06.228109 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:06.228086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:09:14.366794 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.366757    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"]
Sep 11 18:09:14.371097 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.371070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.374904 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.374883    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f8f6a0-serving-cert\""
Sep 11 18:09:14.390040 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.390008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"]
Sep 11 18:09:14.454529 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.454478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.454704 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.454554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-metrics-certs\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.454704 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.454603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6xc\" (UniqueName: \"kubernetes.io/projected/6a25010e-7252-4554-9c83-80160cb5cee5-kube-api-access-dk6xc\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.454704 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.454622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25010e-7252-4554-9c83-80160cb5cee5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.556121 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.556083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.556302 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.556135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-metrics-certs\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.556302 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.556186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6xc\" (UniqueName: \"kubernetes.io/projected/6a25010e-7252-4554-9c83-80160cb5cee5-kube-api-access-dk6xc\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.556302 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.556208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25010e-7252-4554-9c83-80160cb5cee5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.556302 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:09:14.556278    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-serving-cert: secret "maas-api-e2e-ipp-b-f8f6a0-serving-cert" not found
Sep 11 18:09:14.556450 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:09:14.556370    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls podName:6a25010e-7252-4554-9c83-80160cb5cee5 nodeName:}" failed. No retries permitted until 2026-09-11 18:09:15.056351096 +0000 UTC m=+1821.819805803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls") pod "maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" (UID: "6a25010e-7252-4554-9c83-80160cb5cee5") : secret "maas-api-e2e-ipp-b-f8f6a0-serving-cert" not found
Sep 11 18:09:14.556896 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.556869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25010e-7252-4554-9c83-80160cb5cee5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.558889 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.558862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-metrics-certs\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:14.567422 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:14.567398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6xc\" (UniqueName: \"kubernetes.io/projected/6a25010e-7252-4554-9c83-80160cb5cee5-kube-api-access-dk6xc\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:15.061273 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:15.061228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:15.063996 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:15.063966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:15.281786 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:15.281750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:15.569228 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:15.569198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"]
Sep 11 18:09:15.569778 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:09:15.569753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a25010e_7252_4554_9c83_80160cb5cee5.slice/crio-b835bda88095e436961f7b5824f4226b11de130f678cc51c9e493b9c9ba5432b WatchSource:0}: Error finding container b835bda88095e436961f7b5824f4226b11de130f678cc51c9e493b9c9ba5432b: Status 404 returned error can't find the container with id b835bda88095e436961f7b5824f4226b11de130f678cc51c9e493b9c9ba5432b
Sep 11 18:09:15.571212 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:15.571194    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:09:15.691003 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:15.690911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" event={"ID":"6a25010e-7252-4554-9c83-80160cb5cee5","Type":"ContainerStarted","Data":"b835bda88095e436961f7b5824f4226b11de130f678cc51c9e493b9c9ba5432b"}
Sep 11 18:09:16.699146 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:16.699109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" event={"ID":"6a25010e-7252-4554-9c83-80160cb5cee5","Type":"ContainerStarted","Data":"d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392"}
Sep 11 18:09:16.699594 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:16.699206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:09:16.739061 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:16.739001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" podStartSLOduration=2.292173785 podStartE2EDuration="2.738982923s" totalImagesPullingTime="446.809138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:16.735273623 +0000 UTC m=+1823.498728349" watchObservedRunningTime="2026-09-11 18:09:16.738982923 +0000 UTC m=+1823.502437678"
Sep 11 18:09:22.710297 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:09:22.710264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:10:28.261019 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:28.260931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t_4dab35a1-6f01-4c5c-ace7-24cc82d8904d/payload-processing/0.log"
Sep 11 18:10:37.381617 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.381578    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"]
Sep 11 18:10:37.385521 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.385481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.388964 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.388946    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2db21a-dockercfg-q6jj2\""
Sep 11 18:10:37.389519 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.389502    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2db21a\""
Sep 11 18:10:37.403975 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.403948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"]
Sep 11 18:10:37.433638 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.433603    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"]
Sep 11 18:10:37.437165 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.437149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.450670 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.450643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"]
Sep 11 18:10:37.508621 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.508590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669fe089-2347-4fa2-82a8-a0864b703e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.508804 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.508636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmlkf\" (UniqueName: \"kubernetes.io/projected/876cfa69-7bdd-4fde-b330-47cb94782efb-kube-api-access-dmlkf\") pod \"payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.508804 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.508735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876cfa69-7bdd-4fde-b330-47cb94782efb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.508804 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.508795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm277\" (UniqueName: \"kubernetes.io/projected/669fe089-2347-4fa2-82a8-a0864b703e11-kube-api-access-pm277\") pod \"payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.610108 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.610064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876cfa69-7bdd-4fde-b330-47cb94782efb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.610324 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.610157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm277\" (UniqueName: \"kubernetes.io/projected/669fe089-2347-4fa2-82a8-a0864b703e11-kube-api-access-pm277\") pod \"payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.610324 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.610191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669fe089-2347-4fa2-82a8-a0864b703e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.610324 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.610230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmlkf\" (UniqueName: \"kubernetes.io/projected/876cfa69-7bdd-4fde-b330-47cb94782efb-kube-api-access-dmlkf\") pod \"payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.610900 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.610875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669fe089-2347-4fa2-82a8-a0864b703e11-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.610900 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.610892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876cfa69-7bdd-4fde-b330-47cb94782efb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.619861 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.619836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm277\" (UniqueName: \"kubernetes.io/projected/669fe089-2347-4fa2-82a8-a0864b703e11-kube-api-access-pm277\") pod \"payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.620233 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.620215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmlkf\" (UniqueName: \"kubernetes.io/projected/876cfa69-7bdd-4fde-b330-47cb94782efb-kube-api-access-dmlkf\") pod \"payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.695736 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.695648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:37.747971 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.747939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:37.978697 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:37.978670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"]
Sep 11 18:10:37.979470 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:10:37.979442    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876cfa69_7bdd_4fde_b330_47cb94782efb.slice/crio-b02b8164c42856133cb22c9620185647f692300cc58bbe776b4d39bdac302fc1 WatchSource:0}: Error finding container b02b8164c42856133cb22c9620185647f692300cc58bbe776b4d39bdac302fc1: Status 404 returned error can't find the container with id b02b8164c42856133cb22c9620185647f692300cc58bbe776b4d39bdac302fc1
Sep 11 18:10:38.046081 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.046053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"]
Sep 11 18:10:38.049194 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:10:38.049167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669fe089_2347_4fa2_82a8_a0864b703e11.slice/crio-90b0d08baccba08993535c3a5cbd0be0a9bf4ce4eb086e989789de9f1773a627 WatchSource:0}: Error finding container 90b0d08baccba08993535c3a5cbd0be0a9bf4ce4eb086e989789de9f1773a627: Status 404 returned error can't find the container with id 90b0d08baccba08993535c3a5cbd0be0a9bf4ce4eb086e989789de9f1773a627
Sep 11 18:10:38.197384 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.197341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s" event={"ID":"876cfa69-7bdd-4fde-b330-47cb94782efb","Type":"ContainerStarted","Data":"aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96"}
Sep 11 18:10:38.197601 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.197422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s" event={"ID":"876cfa69-7bdd-4fde-b330-47cb94782efb","Type":"ContainerStarted","Data":"b02b8164c42856133cb22c9620185647f692300cc58bbe776b4d39bdac302fc1"}
Sep 11 18:10:38.197601 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.197469    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:38.199429 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.199398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72" event={"ID":"669fe089-2347-4fa2-82a8-a0864b703e11","Type":"ContainerStarted","Data":"6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0"}
Sep 11 18:10:38.199585 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.199450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72" event={"ID":"669fe089-2347-4fa2-82a8-a0864b703e11","Type":"ContainerStarted","Data":"90b0d08baccba08993535c3a5cbd0be0a9bf4ce4eb086e989789de9f1773a627"}
Sep 11 18:10:38.199585 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.199479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:38.228363 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:38.228252    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s" podStartSLOduration=1.2282363059999999 podStartE2EDuration="1.228236306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:38.225235875 +0000 UTC m=+1904.988690631" watchObservedRunningTime="2026-09-11 18:10:38.228236306 +0000 UTC m=+1904.991691043"
Sep 11 18:10:49.208235 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:49.208201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:10:49.208751 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:49.208284    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:10:49.247307 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:10:49.247246    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72" podStartSLOduration=12.247229871 podStartE2EDuration="12.247229871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:38.249178062 +0000 UTC m=+1905.012632789" watchObservedRunningTime="2026-09-11 18:10:49.247229871 +0000 UTC m=+1916.010684599"
Sep 11 18:11:14.546956 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.546913    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"]
Sep 11 18:11:14.547572 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.547176    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s" podUID="876cfa69-7bdd-4fde-b330-47cb94782efb" containerName="payload-processing" containerID="cri-o://aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96" gracePeriod=30
Sep 11 18:11:14.551008 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.550976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"]
Sep 11 18:11:14.551229 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.551205    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72" podUID="669fe089-2347-4fa2-82a8-a0864b703e11" containerName="payload-pre-processing" containerID="cri-o://6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0" gracePeriod=30
Sep 11 18:11:14.853102 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.853075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:11:14.856332 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.856307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:11:14.937918 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.937386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmlkf\" (UniqueName: \"kubernetes.io/projected/876cfa69-7bdd-4fde-b330-47cb94782efb-kube-api-access-dmlkf\") pod \"876cfa69-7bdd-4fde-b330-47cb94782efb\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") "
Sep 11 18:11:14.937918 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.937574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876cfa69-7bdd-4fde-b330-47cb94782efb-plugins-config-volume\") pod \"876cfa69-7bdd-4fde-b330-47cb94782efb\" (UID: \"876cfa69-7bdd-4fde-b330-47cb94782efb\") "
Sep 11 18:11:14.937918 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.937626    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm277\" (UniqueName: \"kubernetes.io/projected/669fe089-2347-4fa2-82a8-a0864b703e11-kube-api-access-pm277\") pod \"669fe089-2347-4fa2-82a8-a0864b703e11\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") "
Sep 11 18:11:14.937918 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.937657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669fe089-2347-4fa2-82a8-a0864b703e11-plugins-config-volume\") pod \"669fe089-2347-4fa2-82a8-a0864b703e11\" (UID: \"669fe089-2347-4fa2-82a8-a0864b703e11\") "
Sep 11 18:11:14.939033 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.938858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876cfa69-7bdd-4fde-b330-47cb94782efb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "876cfa69-7bdd-4fde-b330-47cb94782efb" (UID: "876cfa69-7bdd-4fde-b330-47cb94782efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:11:14.939182 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.939079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669fe089-2347-4fa2-82a8-a0864b703e11-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "669fe089-2347-4fa2-82a8-a0864b703e11" (UID: "669fe089-2347-4fa2-82a8-a0864b703e11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:11:14.939390 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.939369    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876cfa69-7bdd-4fde-b330-47cb94782efb-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:14.939787 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.939748    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/669fe089-2347-4fa2-82a8-a0864b703e11-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:14.941269 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.941237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876cfa69-7bdd-4fde-b330-47cb94782efb-kube-api-access-dmlkf" (OuterVolumeSpecName: "kube-api-access-dmlkf") pod "876cfa69-7bdd-4fde-b330-47cb94782efb" (UID: "876cfa69-7bdd-4fde-b330-47cb94782efb"). InnerVolumeSpecName "kube-api-access-dmlkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:14.941347 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:14.941255    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669fe089-2347-4fa2-82a8-a0864b703e11-kube-api-access-pm277" (OuterVolumeSpecName: "kube-api-access-pm277") pod "669fe089-2347-4fa2-82a8-a0864b703e11" (UID: "669fe089-2347-4fa2-82a8-a0864b703e11"). InnerVolumeSpecName "kube-api-access-pm277". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:15.040461 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.040426    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmlkf\" (UniqueName: \"kubernetes.io/projected/876cfa69-7bdd-4fde-b330-47cb94782efb-kube-api-access-dmlkf\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:15.040461 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.040453    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm277\" (UniqueName: \"kubernetes.io/projected/669fe089-2347-4fa2-82a8-a0864b703e11-kube-api-access-pm277\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:15.423484 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.423440    2563 generic.go:356] "Generic (PLEG): container finished" podID="669fe089-2347-4fa2-82a8-a0864b703e11" containerID="6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0" exitCode=0
Sep 11 18:11:15.423692 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.423540    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"
Sep 11 18:11:15.423692 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.423539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72" event={"ID":"669fe089-2347-4fa2-82a8-a0864b703e11","Type":"ContainerDied","Data":"6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0"}
Sep 11 18:11:15.423692 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.423625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72" event={"ID":"669fe089-2347-4fa2-82a8-a0864b703e11","Type":"ContainerDied","Data":"90b0d08baccba08993535c3a5cbd0be0a9bf4ce4eb086e989789de9f1773a627"}
Sep 11 18:11:15.423692 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.423661    2563 scope.go:117] "RemoveContainer" containerID="6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0"
Sep 11 18:11:15.425046 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.425017    2563 generic.go:356] "Generic (PLEG): container finished" podID="876cfa69-7bdd-4fde-b330-47cb94782efb" containerID="aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96" exitCode=0
Sep 11 18:11:15.425046 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.425042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s" event={"ID":"876cfa69-7bdd-4fde-b330-47cb94782efb","Type":"ContainerDied","Data":"aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96"}
Sep 11 18:11:15.425234 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.425075    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"
Sep 11 18:11:15.425234 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.425083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s" event={"ID":"876cfa69-7bdd-4fde-b330-47cb94782efb","Type":"ContainerDied","Data":"b02b8164c42856133cb22c9620185647f692300cc58bbe776b4d39bdac302fc1"}
Sep 11 18:11:15.434667 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.434643    2563 scope.go:117] "RemoveContainer" containerID="6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0"
Sep 11 18:11:15.435070 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:11:15.434973    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0\": container with ID starting with 6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0 not found: ID does not exist" containerID="6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0"
Sep 11 18:11:15.435070 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.435005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0"} err="failed to get container status \"6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0\": rpc error: code = NotFound desc = could not find container \"6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0\": container with ID starting with 6ce4e915f8c7097b126d5d2ddb2cfe28d84e4c2ee86bffc602fe5db2c02132e0 not found: ID does not exist"
Sep 11 18:11:15.435070 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.435026    2563 scope.go:117] "RemoveContainer" containerID="aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96"
Sep 11 18:11:15.444381 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.444362    2563 scope.go:117] "RemoveContainer" containerID="aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96"
Sep 11 18:11:15.444754 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:11:15.444707    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96\": container with ID starting with aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96 not found: ID does not exist" containerID="aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96"
Sep 11 18:11:15.444848 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.444765    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96"} err="failed to get container status \"aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96\": rpc error: code = NotFound desc = could not find container \"aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96\": container with ID starting with aac28a49bcfe13871424dfa1a5aff9d355ef66999c7f3e508fd66a77a4419f96 not found: ID does not exist"
Sep 11 18:11:15.450876 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.450845    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"]
Sep 11 18:11:15.458546 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.458522    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2db21a-664668ff-9gs8s"]
Sep 11 18:11:15.488333 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.488298    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"]
Sep 11 18:11:15.490776 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.490753    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2db21a-64d6cdb894-dbz72"]
Sep 11 18:11:15.783162 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.783076    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669fe089-2347-4fa2-82a8-a0864b703e11" path="/var/lib/kubelet/pods/669fe089-2347-4fa2-82a8-a0864b703e11/volumes"
Sep 11 18:11:15.783663 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:15.783641    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="876cfa69-7bdd-4fde-b330-47cb94782efb" path="/var/lib/kubelet/pods/876cfa69-7bdd-4fde-b330-47cb94782efb/volumes"
Sep 11 18:11:41.186179 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.186116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"]
Sep 11 18:11:41.186731 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.186432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t" podUID="4dab35a1-6f01-4c5c-ace7-24cc82d8904d" containerName="payload-processing" containerID="cri-o://a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd" gracePeriod=30
Sep 11 18:11:41.190762 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.190724    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"]
Sep 11 18:11:41.191318 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.191237    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf" podUID="99d537b1-a65a-4966-89db-65073132255b" containerName="payload-pre-processing" containerID="cri-o://707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9" gracePeriod=30
Sep 11 18:11:41.490137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.490081    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:11:41.494646 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.494618    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:11:41.579146 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.579105    2563 generic.go:356] "Generic (PLEG): container finished" podID="99d537b1-a65a-4966-89db-65073132255b" containerID="707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9" exitCode=0
Sep 11 18:11:41.579146 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.579149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"
Sep 11 18:11:41.579425 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.579188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf" event={"ID":"99d537b1-a65a-4966-89db-65073132255b","Type":"ContainerDied","Data":"707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9"}
Sep 11 18:11:41.579425 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.579252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf" event={"ID":"99d537b1-a65a-4966-89db-65073132255b","Type":"ContainerDied","Data":"00034f924e9cff1194003c00fd24e7f9ca10d00cdb8b483b17cf916fd6ce61f2"}
Sep 11 18:11:41.579425 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.579281    2563 scope.go:117] "RemoveContainer" containerID="707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9"
Sep 11 18:11:41.580906 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.580883    2563 generic.go:356] "Generic (PLEG): container finished" podID="4dab35a1-6f01-4c5c-ace7-24cc82d8904d" containerID="a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd" exitCode=0
Sep 11 18:11:41.580997 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.580921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t" event={"ID":"4dab35a1-6f01-4c5c-ace7-24cc82d8904d","Type":"ContainerDied","Data":"a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd"}
Sep 11 18:11:41.580997 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.580935    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"
Sep 11 18:11:41.580997 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.580960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t" event={"ID":"4dab35a1-6f01-4c5c-ace7-24cc82d8904d","Type":"ContainerDied","Data":"26dc64457569fdc7e6c1485eaea82dd0ac58d5468f9ee2b99df60ea49798eca6"}
Sep 11 18:11:41.589201 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.589181    2563 scope.go:117] "RemoveContainer" containerID="707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9"
Sep 11 18:11:41.589578 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:11:41.589554    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9\": container with ID starting with 707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9 not found: ID does not exist" containerID="707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9"
Sep 11 18:11:41.589650 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.589590    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9"} err="failed to get container status \"707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9\": rpc error: code = NotFound desc = could not find container \"707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9\": container with ID starting with 707798c6b9097977e63693f9b9e02b182a48245f08666c7c5b51875fd4f41de9 not found: ID does not exist"
Sep 11 18:11:41.589650 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.589613    2563 scope.go:117] "RemoveContainer" containerID="a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd"
Sep 11 18:11:41.598694 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.598666    2563 scope.go:117] "RemoveContainer" containerID="a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd"
Sep 11 18:11:41.599026 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:11:41.599006    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd\": container with ID starting with a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd not found: ID does not exist" containerID="a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd"
Sep 11 18:11:41.599102 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.599033    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd"} err="failed to get container status \"a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd\": rpc error: code = NotFound desc = could not find container \"a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd\": container with ID starting with a177b573ab6c8692549861f0e29ad5afd3d6d98dca9798f51515aed7090663dd not found: ID does not exist"
Sep 11 18:11:41.613538 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.613471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d537b1-a65a-4966-89db-65073132255b-plugins-config-volume\") pod \"99d537b1-a65a-4966-89db-65073132255b\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") "
Sep 11 18:11:41.613721 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.613568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-plugins-config-volume\") pod \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") "
Sep 11 18:11:41.613721 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.613601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vhtn\" (UniqueName: \"kubernetes.io/projected/99d537b1-a65a-4966-89db-65073132255b-kube-api-access-5vhtn\") pod \"99d537b1-a65a-4966-89db-65073132255b\" (UID: \"99d537b1-a65a-4966-89db-65073132255b\") "
Sep 11 18:11:41.613721 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.613683    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfqvc\" (UniqueName: \"kubernetes.io/projected/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-kube-api-access-qfqvc\") pod \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\" (UID: \"4dab35a1-6f01-4c5c-ace7-24cc82d8904d\") "
Sep 11 18:11:41.614070 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.614026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d537b1-a65a-4966-89db-65073132255b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99d537b1-a65a-4966-89db-65073132255b" (UID: "99d537b1-a65a-4966-89db-65073132255b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:11:41.614249 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.614222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dab35a1-6f01-4c5c-ace7-24cc82d8904d" (UID: "4dab35a1-6f01-4c5c-ace7-24cc82d8904d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:11:41.616091 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.616068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-kube-api-access-qfqvc" (OuterVolumeSpecName: "kube-api-access-qfqvc") pod "4dab35a1-6f01-4c5c-ace7-24cc82d8904d" (UID: "4dab35a1-6f01-4c5c-ace7-24cc82d8904d"). InnerVolumeSpecName "kube-api-access-qfqvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:41.616171 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.616113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d537b1-a65a-4966-89db-65073132255b-kube-api-access-5vhtn" (OuterVolumeSpecName: "kube-api-access-5vhtn") pod "99d537b1-a65a-4966-89db-65073132255b" (UID: "99d537b1-a65a-4966-89db-65073132255b"). InnerVolumeSpecName "kube-api-access-5vhtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:11:41.715110 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.715010    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d537b1-a65a-4966-89db-65073132255b-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:41.715110 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.715046    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-plugins-config-volume\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:41.715110 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.715059    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vhtn\" (UniqueName: \"kubernetes.io/projected/99d537b1-a65a-4966-89db-65073132255b-kube-api-access-5vhtn\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:41.715110 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.715070    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqvc\" (UniqueName: \"kubernetes.io/projected/4dab35a1-6f01-4c5c-ace7-24cc82d8904d-kube-api-access-qfqvc\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:11:41.901107 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.901071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"]
Sep 11 18:11:41.907842 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.907814    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-64600a-6b57f45b8b-jbgvf"]
Sep 11 18:11:41.922063 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.922028    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"]
Sep 11 18:11:41.927641 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:41.927615    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-64600a-84d7d4868c-dl26t"]
Sep 11 18:11:43.782288 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:43.782250    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dab35a1-6f01-4c5c-ace7-24cc82d8904d" path="/var/lib/kubelet/pods/4dab35a1-6f01-4c5c-ace7-24cc82d8904d/volumes"
Sep 11 18:11:43.782711 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:11:43.782648    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d537b1-a65a-4966-89db-65073132255b" path="/var/lib/kubelet/pods/99d537b1-a65a-4966-89db-65073132255b/volumes"
Sep 11 18:12:30.205893 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:30.205862    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"]
Sep 11 18:12:30.206424 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:30.206095    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy" containerID="cri-o://924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23" gracePeriod=30
Sep 11 18:12:35.468347 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.468279    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:12:35.560635 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-credential-socket\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560635 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560641    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-podinfo\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560897 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ea70a8-e42b-49d3-8672-179bdee19e2c-istiod-ca-cert\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560897 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-certs\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560897 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-data\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560897 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-envoy\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560897 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8db4\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-kube-api-access-s8db4\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.560897 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-socket\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.561210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560901    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-token\") pod \"85ea70a8-e42b-49d3-8672-179bdee19e2c\" (UID: \"85ea70a8-e42b-49d3-8672-179bdee19e2c\") "
Sep 11 18:12:35.561210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.560911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:12:35.561210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:12:35.561210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:12:35.561210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ea70a8-e42b-49d3-8672-179bdee19e2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:35.561464 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:12:35.561464 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561420    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.561464 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561440    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ea70a8-e42b-49d3-8672-179bdee19e2c-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.561464 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561455    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.561664 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561468    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.561664 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.561480    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.563144 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.563115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:12:35.563269 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.563218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:12:35.563659 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.563631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:35.563754 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.563657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-kube-api-access-s8db4" (OuterVolumeSpecName: "kube-api-access-s8db4") pod "85ea70a8-e42b-49d3-8672-179bdee19e2c" (UID: "85ea70a8-e42b-49d3-8672-179bdee19e2c"). InnerVolumeSpecName "kube-api-access-s8db4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:35.662847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.662807    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.662847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.662841    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.662847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.662849    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ea70a8-e42b-49d3-8672-179bdee19e2c-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.662847 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.662859    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8db4\" (UniqueName: \"kubernetes.io/projected/85ea70a8-e42b-49d3-8672-179bdee19e2c-kube-api-access-s8db4\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:35.905010 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.904972    2563 generic.go:356] "Generic (PLEG): container finished" podID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerID="924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23" exitCode=0
Sep 11 18:12:35.905187 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.905027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"
Sep 11 18:12:35.905187 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.905050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" event={"ID":"85ea70a8-e42b-49d3-8672-179bdee19e2c","Type":"ContainerDied","Data":"924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23"}
Sep 11 18:12:35.905187 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.905107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74" event={"ID":"85ea70a8-e42b-49d3-8672-179bdee19e2c","Type":"ContainerDied","Data":"320708d6bdd13379e7fd8dd030ea05dc888721ab240b2c102b654499a114398d"}
Sep 11 18:12:35.905187 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.905130    2563 scope.go:117] "RemoveContainer" containerID="924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23"
Sep 11 18:12:35.914793 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.914753    2563 scope.go:117] "RemoveContainer" containerID="a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"
Sep 11 18:12:35.924186 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.924162    2563 scope.go:117] "RemoveContainer" containerID="924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23"
Sep 11 18:12:35.924605 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:12:35.924577    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23\": container with ID starting with 924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23 not found: ID does not exist" containerID="924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23"
Sep 11 18:12:35.924728 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.924618    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23"} err="failed to get container status \"924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23\": rpc error: code = NotFound desc = could not find container \"924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23\": container with ID starting with 924378688f82e2a749747e83cb2e1cd21311d542dff835eeb5464ed1d77c1c23 not found: ID does not exist"
Sep 11 18:12:35.924728 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.924644    2563 scope.go:117] "RemoveContainer" containerID="a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"
Sep 11 18:12:35.924960 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:12:35.924942    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d\": container with ID starting with a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d not found: ID does not exist" containerID="a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"
Sep 11 18:12:35.925007 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.924966    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d"} err="failed to get container status \"a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d\": rpc error: code = NotFound desc = could not find container \"a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d\": container with ID starting with a8c20c0beee342e2e60e8448a286348cd48254748ab22ead56dae342bafb4f1d not found: ID does not exist"
Sep 11 18:12:35.927200 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.927177    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"]
Sep 11 18:12:35.933150 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:35.933122    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-64600a-openshift-default-5bb9cb9d96-khz74"]
Sep 11 18:12:37.782537 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:37.782479    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" path="/var/lib/kubelet/pods/85ea70a8-e42b-49d3-8672-179bdee19e2c/volumes"
Sep 11 18:12:41.174700 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.174649    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"]
Sep 11 18:12:41.175155 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.174942    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" podUID="6a25010e-7252-4554-9c83-80160cb5cee5" containerName="maas-api" containerID="cri-o://d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392" gracePeriod=30
Sep 11 18:12:41.437533 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.437428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:12:41.521342 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.521293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25010e-7252-4554-9c83-80160cb5cee5-openshift-service-ca\") pod \"6a25010e-7252-4554-9c83-80160cb5cee5\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") "
Sep 11 18:12:41.521569 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.521422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk6xc\" (UniqueName: \"kubernetes.io/projected/6a25010e-7252-4554-9c83-80160cb5cee5-kube-api-access-dk6xc\") pod \"6a25010e-7252-4554-9c83-80160cb5cee5\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") "
Sep 11 18:12:41.521569 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.521483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls\") pod \"6a25010e-7252-4554-9c83-80160cb5cee5\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") "
Sep 11 18:12:41.521569 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.521529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-metrics-certs\") pod \"6a25010e-7252-4554-9c83-80160cb5cee5\" (UID: \"6a25010e-7252-4554-9c83-80160cb5cee5\") "
Sep 11 18:12:41.521781 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.521750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a25010e-7252-4554-9c83-80160cb5cee5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a25010e-7252-4554-9c83-80160cb5cee5" (UID: "6a25010e-7252-4554-9c83-80160cb5cee5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:41.521919 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.521894    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a25010e-7252-4554-9c83-80160cb5cee5-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:41.523863 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.523834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a25010e-7252-4554-9c83-80160cb5cee5-kube-api-access-dk6xc" (OuterVolumeSpecName: "kube-api-access-dk6xc") pod "6a25010e-7252-4554-9c83-80160cb5cee5" (UID: "6a25010e-7252-4554-9c83-80160cb5cee5"). InnerVolumeSpecName "kube-api-access-dk6xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:41.524524 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.524462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a25010e-7252-4554-9c83-80160cb5cee5" (UID: "6a25010e-7252-4554-9c83-80160cb5cee5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:41.524639 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.524519    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a25010e-7252-4554-9c83-80160cb5cee5" (UID: "6a25010e-7252-4554-9c83-80160cb5cee5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:41.622928 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.622888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6xc\" (UniqueName: \"kubernetes.io/projected/6a25010e-7252-4554-9c83-80160cb5cee5-kube-api-access-dk6xc\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:41.622928 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.622922    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:41.622928 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.622931    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a25010e-7252-4554-9c83-80160cb5cee5-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:41.944918 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.944881    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a25010e-7252-4554-9c83-80160cb5cee5" containerID="d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392" exitCode=0
Sep 11 18:12:41.945098 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.944936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"
Sep 11 18:12:41.945098 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.944976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" event={"ID":"6a25010e-7252-4554-9c83-80160cb5cee5","Type":"ContainerDied","Data":"d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392"}
Sep 11 18:12:41.945098 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.945035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6" event={"ID":"6a25010e-7252-4554-9c83-80160cb5cee5","Type":"ContainerDied","Data":"b835bda88095e436961f7b5824f4226b11de130f678cc51c9e493b9c9ba5432b"}
Sep 11 18:12:41.945098 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.945062    2563 scope.go:117] "RemoveContainer" containerID="d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392"
Sep 11 18:12:41.954401 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.954232    2563 scope.go:117] "RemoveContainer" containerID="d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392"
Sep 11 18:12:41.954521 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:12:41.954503    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392\": container with ID starting with d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392 not found: ID does not exist" containerID="d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392"
Sep 11 18:12:41.954603 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.954537    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392"} err="failed to get container status \"d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392\": rpc error: code = NotFound desc = could not find container \"d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392\": container with ID starting with d05ee0b2544f3077b192267e0e6302ac0b53d14036ee7ae2559092730585c392 not found: ID does not exist"
Sep 11 18:12:41.971337 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.971306    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"]
Sep 11 18:12:41.977904 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:41.977877    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f8f6a0-675cfcc9b9-fx6l6"]
Sep 11 18:12:43.782099 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:43.782070    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a25010e-7252-4554-9c83-80160cb5cee5" path="/var/lib/kubelet/pods/6a25010e-7252-4554-9c83-80160cb5cee5/volumes"
Sep 11 18:12:59.522200 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.522159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"]
Sep 11 18:12:59.522694 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.522513    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" podUID="cae5fbd5-471a-4424-b922-9dcc423b072e" containerName="maas-api" containerID="cri-o://f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64" gracePeriod=30
Sep 11 18:12:59.780413 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.780340    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 18:12:59.803656 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.803617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae5fbd5-471a-4424-b922-9dcc423b072e-openshift-service-ca\") pod \"cae5fbd5-471a-4424-b922-9dcc423b072e\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") "
Sep 11 18:12:59.803846 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.803676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkvkl\" (UniqueName: \"kubernetes.io/projected/cae5fbd5-471a-4424-b922-9dcc423b072e-kube-api-access-hkvkl\") pod \"cae5fbd5-471a-4424-b922-9dcc423b072e\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") "
Sep 11 18:12:59.803846 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.803722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-metrics-certs\") pod \"cae5fbd5-471a-4424-b922-9dcc423b072e\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") "
Sep 11 18:12:59.803846 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.803800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls\") pod \"cae5fbd5-471a-4424-b922-9dcc423b072e\" (UID: \"cae5fbd5-471a-4424-b922-9dcc423b072e\") "
Sep 11 18:12:59.804120 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.804094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae5fbd5-471a-4424-b922-9dcc423b072e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cae5fbd5-471a-4424-b922-9dcc423b072e" (UID: "cae5fbd5-471a-4424-b922-9dcc423b072e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:59.806205 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.806175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cae5fbd5-471a-4424-b922-9dcc423b072e" (UID: "cae5fbd5-471a-4424-b922-9dcc423b072e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:59.806308 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.806207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae5fbd5-471a-4424-b922-9dcc423b072e-kube-api-access-hkvkl" (OuterVolumeSpecName: "kube-api-access-hkvkl") pod "cae5fbd5-471a-4424-b922-9dcc423b072e" (UID: "cae5fbd5-471a-4424-b922-9dcc423b072e"). InnerVolumeSpecName "kube-api-access-hkvkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:59.806347 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.806306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cae5fbd5-471a-4424-b922-9dcc423b072e" (UID: "cae5fbd5-471a-4424-b922-9dcc423b072e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:59.905382 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.905338    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:59.905382 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.905380    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae5fbd5-471a-4424-b922-9dcc423b072e-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:59.905666 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.905426    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae5fbd5-471a-4424-b922-9dcc423b072e-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:12:59.905666 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:12:59.905441    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkvkl\" (UniqueName: \"kubernetes.io/projected/cae5fbd5-471a-4424-b922-9dcc423b072e-kube-api-access-hkvkl\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:00.043683 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.043588    2563 generic.go:356] "Generic (PLEG): container finished" podID="cae5fbd5-471a-4424-b922-9dcc423b072e" containerID="f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64" exitCode=0
Sep 11 18:13:00.043683 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.043636    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"
Sep 11 18:13:00.043865 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.043672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" event={"ID":"cae5fbd5-471a-4424-b922-9dcc423b072e","Type":"ContainerDied","Data":"f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64"}
Sep 11 18:13:00.043865 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.043732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb" event={"ID":"cae5fbd5-471a-4424-b922-9dcc423b072e","Type":"ContainerDied","Data":"b226405ae7a29dcb3d2063ae68d3722d904d1e63a698928f8398badb771f9147"}
Sep 11 18:13:00.043865 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.043753    2563 scope.go:117] "RemoveContainer" containerID="f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64"
Sep 11 18:13:00.054027 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.054006    2563 scope.go:117] "RemoveContainer" containerID="f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64"
Sep 11 18:13:00.054349 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:13:00.054330    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64\": container with ID starting with f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64 not found: ID does not exist" containerID="f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64"
Sep 11 18:13:00.054424 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.054358    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64"} err="failed to get container status \"f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64\": rpc error: code = NotFound desc = could not find container \"f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64\": container with ID starting with f3f24aff6e77c119480d7eedec8c0864a556d838e12f75dc3885066711d82a64 not found: ID does not exist"
Sep 11 18:13:00.067646 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.067621    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"]
Sep 11 18:13:00.072478 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:00.072451    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a18ac6-59998c65d8-m96hb"]
Sep 11 18:13:01.782179 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:01.782147    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae5fbd5-471a-4424-b922-9dcc423b072e" path="/var/lib/kubelet/pods/cae5fbd5-471a-4424-b922-9dcc423b072e/volumes"
Sep 11 18:13:06.774124 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:06.774089    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"]
Sep 11 18:13:06.774526 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:06.774326    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" containerID="cri-o://ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9" gracePeriod=30
Sep 11 18:13:09.696304 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:09.696269    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:13:09.696719 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:09.696323    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:13:12.026269 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.026240    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 18:13:12.115396 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.115362    2563 generic.go:356] "Generic (PLEG): container finished" podID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerID="ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9" exitCode=0
Sep 11 18:13:12.115396 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.115402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" event={"ID":"97493b51-0ea8-4ada-9caa-8b6cf3a2815d","Type":"ContainerDied","Data":"ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9"}
Sep 11 18:13:12.115649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.115415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"
Sep 11 18:13:12.115649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.115447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh" event={"ID":"97493b51-0ea8-4ada-9caa-8b6cf3a2815d","Type":"ContainerDied","Data":"fc8d9318e18f9547e2ab477074f83a533b1122a75fd10fe4673bd29cd8d760de"}
Sep 11 18:13:12.115649 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.115469    2563 scope.go:117] "RemoveContainer" containerID="ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9"
Sep 11 18:13:12.124565 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.124547    2563 scope.go:117] "RemoveContainer" containerID="19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"
Sep 11 18:13:12.127815 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.127791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-certs\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.127940 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.127827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-socket\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.127940 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.127847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-data\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.127940 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.127864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rtjz\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-kube-api-access-9rtjz\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.127940 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.127899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-envoy\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.127950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-podinfo\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-credential-socket\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128084    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istiod-ca-cert\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128134    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-token\") pod \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\" (UID: \"97493b51-0ea8-4ada-9caa-8b6cf3a2815d\") "
Sep 11 18:13:12.128178 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-data" (OuterVolumeSpecName: "istio-data") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:12.128559 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128360    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:12.128559 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:13:12.128660 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128558    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.128660 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128578    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-workload-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.128660 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128591    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-data\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.128660 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128604    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-credential-socket\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.128660 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.128617    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istiod-ca-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.130690 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.130658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-kube-api-access-9rtjz" (OuterVolumeSpecName: "kube-api-access-9rtjz") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "kube-api-access-9rtjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:12.130795 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.130745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-token" (OuterVolumeSpecName: "istio-token") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:12.130867 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.130819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:12.130867 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.130828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97493b51-0ea8-4ada-9caa-8b6cf3a2815d" (UID: "97493b51-0ea8-4ada-9caa-8b6cf3a2815d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:13:12.134288 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.134267    2563 scope.go:117] "RemoveContainer" containerID="ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9"
Sep 11 18:13:12.134688 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:13:12.134663    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9\": container with ID starting with ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9 not found: ID does not exist" containerID="ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9"
Sep 11 18:13:12.134800 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.134700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9"} err="failed to get container status \"ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9\": rpc error: code = NotFound desc = could not find container \"ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9\": container with ID starting with ee4af58776af939d8ae6602acc46780bed74271e440cd53dd52a2a7d961185b9 not found: ID does not exist"
Sep 11 18:13:12.134800 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.134725    2563 scope.go:117] "RemoveContainer" containerID="19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"
Sep 11 18:13:12.135070 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:13:12.135051    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596\": container with ID starting with 19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596 not found: ID does not exist" containerID="19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"
Sep 11 18:13:12.135139 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.135088    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596"} err="failed to get container status \"19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596\": rpc error: code = NotFound desc = could not find container \"19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596\": container with ID starting with 19fe7b2860f0b8516775dade448eae37925c4d140aca710b9607c6527d0da596 not found: ID does not exist"
Sep 11 18:13:12.229433 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.229391    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-podinfo\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.229433 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.229424    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-token\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.229433 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.229435    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtjz\" (UniqueName: \"kubernetes.io/projected/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-kube-api-access-9rtjz\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.229433 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.229443    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97493b51-0ea8-4ada-9caa-8b6cf3a2815d-istio-envoy\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:12.443956 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.443922    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"]
Sep 11 18:13:12.449142 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:12.449109    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a18ac6-openshift-default-59576d9ffb-2h7kh"]
Sep 11 18:13:13.782978 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:13.782942    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" path="/var/lib/kubelet/pods/97493b51-0ea8-4ada-9caa-8b6cf3a2815d/volumes"
Sep 11 18:13:17.695303 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:17.695224    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"]
Sep 11 18:13:17.695797 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:17.695469    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v" podUID="9c7ab737-7caa-426b-a271-8650aa3a69b8" containerName="maas-api" containerID="cri-o://ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe" gracePeriod=30
Sep 11 18:13:17.946137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:17.946065    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 18:13:18.087835 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.087799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ab737-7caa-426b-a271-8650aa3a69b8-openshift-service-ca\") pod \"9c7ab737-7caa-426b-a271-8650aa3a69b8\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") "
Sep 11 18:13:18.088044 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.087971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzvc7\" (UniqueName: \"kubernetes.io/projected/9c7ab737-7caa-426b-a271-8650aa3a69b8-kube-api-access-bzvc7\") pod \"9c7ab737-7caa-426b-a271-8650aa3a69b8\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") "
Sep 11 18:13:18.088044 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.088023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-maas-api-tls\") pod \"9c7ab737-7caa-426b-a271-8650aa3a69b8\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") "
Sep 11 18:13:18.088164 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.088062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-metrics-certs\") pod \"9c7ab737-7caa-426b-a271-8650aa3a69b8\" (UID: \"9c7ab737-7caa-426b-a271-8650aa3a69b8\") "
Sep 11 18:13:18.088222 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.088157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7ab737-7caa-426b-a271-8650aa3a69b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c7ab737-7caa-426b-a271-8650aa3a69b8" (UID: "9c7ab737-7caa-426b-a271-8650aa3a69b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:13:18.088399 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.088383    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7ab737-7caa-426b-a271-8650aa3a69b8-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:18.090308 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.090281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c7ab737-7caa-426b-a271-8650aa3a69b8" (UID: "9c7ab737-7caa-426b-a271-8650aa3a69b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:18.090452 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.090433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7ab737-7caa-426b-a271-8650aa3a69b8-kube-api-access-bzvc7" (OuterVolumeSpecName: "kube-api-access-bzvc7") pod "9c7ab737-7caa-426b-a271-8650aa3a69b8" (UID: "9c7ab737-7caa-426b-a271-8650aa3a69b8"). InnerVolumeSpecName "kube-api-access-bzvc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:18.090533 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.090445    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c7ab737-7caa-426b-a271-8650aa3a69b8" (UID: "9c7ab737-7caa-426b-a271-8650aa3a69b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:18.156884 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.156848    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c7ab737-7caa-426b-a271-8650aa3a69b8" containerID="ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe" exitCode=0
Sep 11 18:13:18.157058 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.156893    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"
Sep 11 18:13:18.157058 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.156929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v" event={"ID":"9c7ab737-7caa-426b-a271-8650aa3a69b8","Type":"ContainerDied","Data":"ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe"}
Sep 11 18:13:18.157058 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.156979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v" event={"ID":"9c7ab737-7caa-426b-a271-8650aa3a69b8","Type":"ContainerDied","Data":"52597a590a1c2c379822b681a8769f339e9e606d9d6e3b9c89ccba8713088ae7"}
Sep 11 18:13:18.157058 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.157002    2563 scope.go:117] "RemoveContainer" containerID="ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe"
Sep 11 18:13:18.166604 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.166580    2563 scope.go:117] "RemoveContainer" containerID="ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe"
Sep 11 18:13:18.166964 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:13:18.166938    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe\": container with ID starting with ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe not found: ID does not exist" containerID="ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe"
Sep 11 18:13:18.167047 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.166972    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe"} err="failed to get container status \"ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe\": rpc error: code = NotFound desc = could not find container \"ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe\": container with ID starting with ae9026dc97935f890cd31fa76735f80a767aa29c9253d30022563f50c5ba91fe not found: ID does not exist"
Sep 11 18:13:18.181833 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.181801    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"]
Sep 11 18:13:18.185639 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.185612    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1bf937-54bbb4f6d7-wfx2v"]
Sep 11 18:13:18.189579 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.189549    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:18.189579 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.189582    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvc7\" (UniqueName: \"kubernetes.io/projected/9c7ab737-7caa-426b-a271-8650aa3a69b8-kube-api-access-bzvc7\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:18.189696 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:18.189592    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7ab737-7caa-426b-a271-8650aa3a69b8-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:19.782169 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:19.782131    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c7ab737-7caa-426b-a271-8650aa3a69b8" path="/var/lib/kubelet/pods/9c7ab737-7caa-426b-a271-8650aa3a69b8/volumes"
Sep 11 18:13:27.566791 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.566756    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b57799854-rbrp8"]
Sep 11 18:13:27.567280 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.566961    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b57799854-rbrp8" podUID="39e1ea4e-773d-4637-aaba-bb74230a4d45" containerName="manager" containerID="cri-o://a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9" gracePeriod=30
Sep 11 18:13:27.820778 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.820755    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 18:13:27.981708 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.981670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-webhook-cert\") pod \"39e1ea4e-773d-4637-aaba-bb74230a4d45\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") "
Sep 11 18:13:27.981892 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.981742    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6mxf\" (UniqueName: \"kubernetes.io/projected/39e1ea4e-773d-4637-aaba-bb74230a4d45-kube-api-access-z6mxf\") pod \"39e1ea4e-773d-4637-aaba-bb74230a4d45\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") "
Sep 11 18:13:27.981892 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.981780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-metrics-certs\") pod \"39e1ea4e-773d-4637-aaba-bb74230a4d45\" (UID: \"39e1ea4e-773d-4637-aaba-bb74230a4d45\") "
Sep 11 18:13:27.984068 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.984038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "39e1ea4e-773d-4637-aaba-bb74230a4d45" (UID: "39e1ea4e-773d-4637-aaba-bb74230a4d45"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:27.984215 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.984088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39e1ea4e-773d-4637-aaba-bb74230a4d45" (UID: "39e1ea4e-773d-4637-aaba-bb74230a4d45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:27.984215 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:27.984112    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e1ea4e-773d-4637-aaba-bb74230a4d45-kube-api-access-z6mxf" (OuterVolumeSpecName: "kube-api-access-z6mxf") pod "39e1ea4e-773d-4637-aaba-bb74230a4d45" (UID: "39e1ea4e-773d-4637-aaba-bb74230a4d45"). InnerVolumeSpecName "kube-api-access-z6mxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:28.083532 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.083414    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mxf\" (UniqueName: \"kubernetes.io/projected/39e1ea4e-773d-4637-aaba-bb74230a4d45-kube-api-access-z6mxf\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.083532 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.083451    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.083532 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.083460    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39e1ea4e-773d-4637-aaba-bb74230a4d45-webhook-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.216561 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.216527    2563 generic.go:356] "Generic (PLEG): container finished" podID="39e1ea4e-773d-4637-aaba-bb74230a4d45" containerID="a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9" exitCode=0
Sep 11 18:13:28.216734 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.216565    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-rbrp8"
Sep 11 18:13:28.216734 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.216611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-rbrp8" event={"ID":"39e1ea4e-773d-4637-aaba-bb74230a4d45","Type":"ContainerDied","Data":"a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9"}
Sep 11 18:13:28.216734 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.216662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-rbrp8" event={"ID":"39e1ea4e-773d-4637-aaba-bb74230a4d45","Type":"ContainerDied","Data":"78bf1a7f9ece257746aaa2c38cf630afe961b5b01bb837912383664bd1c58434"}
Sep 11 18:13:28.216734 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.216687    2563 scope.go:117] "RemoveContainer" containerID="a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9"
Sep 11 18:13:28.227002 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.226985    2563 scope.go:117] "RemoveContainer" containerID="a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9"
Sep 11 18:13:28.227282 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:13:28.227264    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9\": container with ID starting with a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9 not found: ID does not exist" containerID="a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9"
Sep 11 18:13:28.227351 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.227290    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9"} err="failed to get container status \"a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9\": rpc error: code = NotFound desc = could not find container \"a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9\": container with ID starting with a20ad7d8dbdf35f224fcefb7d1cd7b0089284f5f0a1daeb8efa7dfc5faa758b9 not found: ID does not exist"
Sep 11 18:13:28.239354 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.239327    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b57799854-rbrp8"]
Sep 11 18:13:28.244239 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:28.244214    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b57799854-rbrp8"]
Sep 11 18:13:29.782630 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:29.782598    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e1ea4e-773d-4637-aaba-bb74230a4d45" path="/var/lib/kubelet/pods/39e1ea4e-773d-4637-aaba-bb74230a4d45/volumes"
Sep 11 18:13:53.979035 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:53.979008    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:13:53.982395 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:13:53.982373    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:14:29.117390 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b57799854-4frrs"]
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d537b1-a65a-4966-89db-65073132255b" containerName="payload-pre-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117750    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d537b1-a65a-4966-89db-65073132255b" containerName="payload-pre-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117763    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117768    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a25010e-7252-4554-9c83-80160cb5cee5" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117781    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a25010e-7252-4554-9c83-80160cb5cee5" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117791    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117802    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876cfa69-7bdd-4fde-b330-47cb94782efb" containerName="payload-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117808    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="876cfa69-7bdd-4fde-b330-47cb94782efb" containerName="payload-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117814    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669fe089-2347-4fa2-82a8-a0864b703e11" containerName="payload-pre-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117820    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="669fe089-2347-4fa2-82a8-a0864b703e11" containerName="payload-pre-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117828    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dab35a1-6f01-4c5c-ace7-24cc82d8904d" containerName="payload-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dab35a1-6f01-4c5c-ace7-24cc82d8904d" containerName="payload-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117839    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae5fbd5-471a-4424-b922-9dcc423b072e" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae5fbd5-471a-4424-b922-9dcc423b072e" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117849    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117854    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117863    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7ab737-7caa-426b-a271-8650aa3a69b8" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117867    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7ab737-7caa-426b-a271-8650aa3a69b8" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117874    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e1ea4e-773d-4637-aaba-bb74230a4d45" containerName="manager"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117878    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e1ea4e-773d-4637-aaba-bb74230a4d45" containerName="manager"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117935    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117942    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="876cfa69-7bdd-4fde-b330-47cb94782efb" containerName="payload-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117949    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dab35a1-6f01-4c5c-ace7-24cc82d8904d" containerName="payload-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117955    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d537b1-a65a-4966-89db-65073132255b" containerName="payload-pre-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117961    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117965    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117970    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="669fe089-2347-4fa2-82a8-a0864b703e11" containerName="payload-pre-processing"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e1ea4e-773d-4637-aaba-bb74230a4d45" containerName="manager"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117978    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a25010e-7252-4554-9c83-80160cb5cee5" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117983    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7ab737-7caa-426b-a271-8650aa3a69b8" containerName="maas-api"
Sep 11 18:14:29.119958 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.117988    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae5fbd5-471a-4424-b922-9dcc423b072e" containerName="maas-api"
Sep 11 18:14:29.120999 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.120981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.126284 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.126261    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tjzmj\""
Sep 11 18:14:29.126446 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.126288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:14:29.126446 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.126388    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:14:29.126648 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.126630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:14:29.141422 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.141397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b57799854-4frrs"]
Sep 11 18:14:29.145031 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.145002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-webhook-cert\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.145153 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.145117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2h8\" (UniqueName: \"kubernetes.io/projected/87c37ba7-a7d5-40ca-bf39-86da06408dd5-kube-api-access-7c2h8\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.145205 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.145151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-metrics-certs\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.245599 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.245564    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2h8\" (UniqueName: \"kubernetes.io/projected/87c37ba7-a7d5-40ca-bf39-86da06408dd5-kube-api-access-7c2h8\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.245799 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.245616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-metrics-certs\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.245799 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.245671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-webhook-cert\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.248395 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.248367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-webhook-cert\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.248527 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.248454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-metrics-certs\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.257336 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.257304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2h8\" (UniqueName: \"kubernetes.io/projected/87c37ba7-a7d5-40ca-bf39-86da06408dd5-kube-api-access-7c2h8\") pod \"maas-controller-5b57799854-4frrs\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") " pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.431769 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.431673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:29.718652 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.717016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b57799854-4frrs"]
Sep 11 18:14:29.719821 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:29.719798    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:14:30.568321 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:30.568285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-4frrs" event={"ID":"87c37ba7-a7d5-40ca-bf39-86da06408dd5","Type":"ContainerStarted","Data":"60e9b8e6c71ab688c816dd38143f0f475f727b5f59d7a688498433ebdacd0e8e"}
Sep 11 18:14:31.575446 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:31.575405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-4frrs" event={"ID":"87c37ba7-a7d5-40ca-bf39-86da06408dd5","Type":"ContainerStarted","Data":"7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7"}
Sep 11 18:14:31.575890 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:31.575560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:31.598245 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:31.598189    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b57799854-4frrs" podStartSLOduration=1.7209117809999999 podStartE2EDuration="2.598169952s" totalImagesPullingTime="877.258171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:31.596079583 +0000 UTC m=+2138.359534312" watchObservedRunningTime="2026-09-11 18:14:31.598169952 +0000 UTC m=+2138.361624678"
Sep 11 18:14:42.582912 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:42.582878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:49.857169 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:49.857081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b57799854-4frrs"]
Sep 11 18:14:49.857650 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:49.857484    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b57799854-4frrs" podUID="87c37ba7-a7d5-40ca-bf39-86da06408dd5" containerName="manager" containerID="cri-o://7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7" gracePeriod=30
Sep 11 18:14:50.108036 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.107964    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:50.235795 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.235760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-metrics-certs\") pod \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") "
Sep 11 18:14:50.235988 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.235849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-webhook-cert\") pod \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") "
Sep 11 18:14:50.235988 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.235881    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c2h8\" (UniqueName: \"kubernetes.io/projected/87c37ba7-a7d5-40ca-bf39-86da06408dd5-kube-api-access-7c2h8\") pod \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\" (UID: \"87c37ba7-a7d5-40ca-bf39-86da06408dd5\") "
Sep 11 18:14:50.238230 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.238196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "87c37ba7-a7d5-40ca-bf39-86da06408dd5" (UID: "87c37ba7-a7d5-40ca-bf39-86da06408dd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:14:50.238379 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.238358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "87c37ba7-a7d5-40ca-bf39-86da06408dd5" (UID: "87c37ba7-a7d5-40ca-bf39-86da06408dd5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:14:50.238441 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.238418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c37ba7-a7d5-40ca-bf39-86da06408dd5-kube-api-access-7c2h8" (OuterVolumeSpecName: "kube-api-access-7c2h8") pod "87c37ba7-a7d5-40ca-bf39-86da06408dd5" (UID: "87c37ba7-a7d5-40ca-bf39-86da06408dd5"). InnerVolumeSpecName "kube-api-access-7c2h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:50.337003 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.336949    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-webhook-cert\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:14:50.337003 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.336994    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2h8\" (UniqueName: \"kubernetes.io/projected/87c37ba7-a7d5-40ca-bf39-86da06408dd5-kube-api-access-7c2h8\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:14:50.337003 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.337006    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87c37ba7-a7d5-40ca-bf39-86da06408dd5-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:14:50.679159 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.679121    2563 generic.go:356] "Generic (PLEG): container finished" podID="87c37ba7-a7d5-40ca-bf39-86da06408dd5" containerID="7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7" exitCode=0
Sep 11 18:14:50.679372 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.679163    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-4frrs"
Sep 11 18:14:50.679372 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.679211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-4frrs" event={"ID":"87c37ba7-a7d5-40ca-bf39-86da06408dd5","Type":"ContainerDied","Data":"7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7"}
Sep 11 18:14:50.679372 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.679268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-4frrs" event={"ID":"87c37ba7-a7d5-40ca-bf39-86da06408dd5","Type":"ContainerDied","Data":"60e9b8e6c71ab688c816dd38143f0f475f727b5f59d7a688498433ebdacd0e8e"}
Sep 11 18:14:50.679372 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.679299    2563 scope.go:117] "RemoveContainer" containerID="7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7"
Sep 11 18:14:50.689275 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.689255    2563 scope.go:117] "RemoveContainer" containerID="7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7"
Sep 11 18:14:50.689633 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:14:50.689613    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7\": container with ID starting with 7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7 not found: ID does not exist" containerID="7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7"
Sep 11 18:14:50.689699 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.689646    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7"} err="failed to get container status \"7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7\": rpc error: code = NotFound desc = could not find container \"7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7\": container with ID starting with 7bc020722a86a2f49aefa46483f764fdbe039e82e28dd4dca94f2fab02530fb7 not found: ID does not exist"
Sep 11 18:14:50.703797 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.703765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b57799854-4frrs"]
Sep 11 18:14:50.708204 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:50.708164    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b57799854-4frrs"]
Sep 11 18:14:51.555320 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.555277    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b57799854-5m5zr"]
Sep 11 18:14:51.555885 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.555865    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy"
Sep 11 18:14:51.555968 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.555888    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ea70a8-e42b-49d3-8672-179bdee19e2c" containerName="istio-proxy"
Sep 11 18:14:51.555968 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.555924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87c37ba7-a7d5-40ca-bf39-86da06408dd5" containerName="manager"
Sep 11 18:14:51.555968 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.555933    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87c37ba7-a7d5-40ca-bf39-86da06408dd5" containerName="manager"
Sep 11 18:14:51.556137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.556054    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87c37ba7-a7d5-40ca-bf39-86da06408dd5" containerName="manager"
Sep 11 18:14:51.556137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.556068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="97493b51-0ea8-4ada-9caa-8b6cf3a2815d" containerName="istio-proxy"
Sep 11 18:14:51.560579 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.560556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.564778 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.564757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:14:51.566330 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.566299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tjzmj\""
Sep 11 18:14:51.566330 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.566323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:14:51.566549 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.566304    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:14:51.570439 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.570412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b57799854-5m5zr"]
Sep 11 18:14:51.649396 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.649352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5afd6465-8760-4ac6-8591-f8d0df862014-webhook-cert\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.649615 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.649426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5afd6465-8760-4ac6-8591-f8d0df862014-metrics-certs\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.649615 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.649456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9s5\" (UniqueName: \"kubernetes.io/projected/5afd6465-8760-4ac6-8591-f8d0df862014-kube-api-access-vk9s5\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.750545 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.750473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5afd6465-8760-4ac6-8591-f8d0df862014-webhook-cert\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.750767 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.750572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5afd6465-8760-4ac6-8591-f8d0df862014-metrics-certs\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.750767 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.750613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9s5\" (UniqueName: \"kubernetes.io/projected/5afd6465-8760-4ac6-8591-f8d0df862014-kube-api-access-vk9s5\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.753307 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.753276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5afd6465-8760-4ac6-8591-f8d0df862014-webhook-cert\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.753430 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.753322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5afd6465-8760-4ac6-8591-f8d0df862014-metrics-certs\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.760869 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.760842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9s5\" (UniqueName: \"kubernetes.io/projected/5afd6465-8760-4ac6-8591-f8d0df862014-kube-api-access-vk9s5\") pod \"maas-controller-5b57799854-5m5zr\" (UID: \"5afd6465-8760-4ac6-8591-f8d0df862014\") " pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:51.781576 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.781547    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87c37ba7-a7d5-40ca-bf39-86da06408dd5" path="/var/lib/kubelet/pods/87c37ba7-a7d5-40ca-bf39-86da06408dd5/volumes"
Sep 11 18:14:51.872940 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:51.872911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:52.151102 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:52.151023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b57799854-5m5zr"]
Sep 11 18:14:52.154371 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:14:52.154333    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afd6465_8760_4ac6_8591_f8d0df862014.slice/crio-61752a87f3aff36c340ca77cf485c3f2bae657a2e6a7e0dabc235c95835efafb WatchSource:0}: Error finding container 61752a87f3aff36c340ca77cf485c3f2bae657a2e6a7e0dabc235c95835efafb: Status 404 returned error can't find the container with id 61752a87f3aff36c340ca77cf485c3f2bae657a2e6a7e0dabc235c95835efafb
Sep 11 18:14:52.693352 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:52.693316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-5m5zr" event={"ID":"5afd6465-8760-4ac6-8591-f8d0df862014","Type":"ContainerStarted","Data":"abf49aaf3013024022c071c22d164621b06bc4f46d131332cd28ca92c0ae8856"}
Sep 11 18:14:52.693870 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:52.693373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b57799854-5m5zr" event={"ID":"5afd6465-8760-4ac6-8591-f8d0df862014","Type":"ContainerStarted","Data":"61752a87f3aff36c340ca77cf485c3f2bae657a2e6a7e0dabc235c95835efafb"}
Sep 11 18:14:52.693870 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:52.693440    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:14:52.719689 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:14:52.719639    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b57799854-5m5zr" podStartSLOduration=1.364951979 podStartE2EDuration="1.719618244s" totalImagesPullingTime="354.666265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:52.716408679 +0000 UTC m=+2159.479863428" watchObservedRunningTime="2026-09-11 18:14:52.719618244 +0000 UTC m=+2159.483072971"
Sep 11 18:15:03.701296 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:15:03.701264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b57799854-5m5zr"
Sep 11 18:18:44.416335 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.416300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"]
Sep 11 18:18:44.419986 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.419964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.437503 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.437446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"]
Sep 11 18:18:44.451561 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.451527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf7dba-0c05-4e79-949e-bc037be20c9e-openshift-service-ca\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.451803 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.451739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdf7dba-0c05-4e79-949e-bc037be20c9e-metrics-certs\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.452164 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.451833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgrh\" (UniqueName: \"kubernetes.io/projected/2cdf7dba-0c05-4e79-949e-bc037be20c9e-kube-api-access-hjgrh\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.452164 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.451972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdf7dba-0c05-4e79-949e-bc037be20c9e-maas-api-tls\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.553549 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.553480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdf7dba-0c05-4e79-949e-bc037be20c9e-metrics-certs\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.553762 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.553563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgrh\" (UniqueName: \"kubernetes.io/projected/2cdf7dba-0c05-4e79-949e-bc037be20c9e-kube-api-access-hjgrh\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.553762 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.553617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdf7dba-0c05-4e79-949e-bc037be20c9e-maas-api-tls\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.553762 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.553652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf7dba-0c05-4e79-949e-bc037be20c9e-openshift-service-ca\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.554290 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.554264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf7dba-0c05-4e79-949e-bc037be20c9e-openshift-service-ca\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.556091 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.556070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cdf7dba-0c05-4e79-949e-bc037be20c9e-maas-api-tls\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.556189 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.556171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cdf7dba-0c05-4e79-949e-bc037be20c9e-metrics-certs\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.572234 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.572208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgrh\" (UniqueName: \"kubernetes.io/projected/2cdf7dba-0c05-4e79-949e-bc037be20c9e-kube-api-access-hjgrh\") pod \"maas-api-6c48469567-jc2gt\" (UID: \"2cdf7dba-0c05-4e79-949e-bc037be20c9e\") " pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:44.730594 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:44.730481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:45.016003 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:45.015974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"]
Sep 11 18:18:45.017898 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:18:45.017869    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdf7dba_0c05_4e79_949e_bc037be20c9e.slice/crio-e00e1d8c9c4de9b15d3e549a050982ebebb73f3c674f06cd75b0476dc9d3eba3 WatchSource:0}: Error finding container e00e1d8c9c4de9b15d3e549a050982ebebb73f3c674f06cd75b0476dc9d3eba3: Status 404 returned error can't find the container with id e00e1d8c9c4de9b15d3e549a050982ebebb73f3c674f06cd75b0476dc9d3eba3
Sep 11 18:18:45.975154 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:45.975105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt" event={"ID":"2cdf7dba-0c05-4e79-949e-bc037be20c9e","Type":"ContainerStarted","Data":"7c494bf7ffad10cfc77e8bd092e1e8dbc49c471e2013886ef0bef85d923c0f12"}
Sep 11 18:18:45.975633 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:45.975164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt" event={"ID":"2cdf7dba-0c05-4e79-949e-bc037be20c9e","Type":"ContainerStarted","Data":"e00e1d8c9c4de9b15d3e549a050982ebebb73f3c674f06cd75b0476dc9d3eba3"}
Sep 11 18:18:45.975633 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:45.975185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:46.019935 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:46.019881    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt" podStartSLOduration=1.659307734 podStartE2EDuration="2.019864026s" totalImagesPullingTime="360.556292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:46.016704608 +0000 UTC m=+2392.780159337" watchObservedRunningTime="2026-09-11 18:18:46.019864026 +0000 UTC m=+2392.783318786"
Sep 11 18:18:51.985506 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:51.985463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c48469567-jc2gt"
Sep 11 18:18:52.049961 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:52.049927    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"]
Sep 11 18:18:52.050170 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:52.050147    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54" podUID="1923587f-4658-4c84-809f-e9984da40fbb" containerName="maas-api" containerID="cri-o://588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484" gracePeriod=30
Sep 11 18:18:53.321871 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.321846    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 18:18:53.443830 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.443734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923587f-4658-4c84-809f-e9984da40fbb-openshift-service-ca\") pod \"1923587f-4658-4c84-809f-e9984da40fbb\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") "
Sep 11 18:18:53.443830 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.443784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-metrics-certs\") pod \"1923587f-4658-4c84-809f-e9984da40fbb\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") "
Sep 11 18:18:53.443830 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.443818    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frgb8\" (UniqueName: \"kubernetes.io/projected/1923587f-4658-4c84-809f-e9984da40fbb-kube-api-access-frgb8\") pod \"1923587f-4658-4c84-809f-e9984da40fbb\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") "
Sep 11 18:18:53.444135 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.443837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls\") pod \"1923587f-4658-4c84-809f-e9984da40fbb\" (UID: \"1923587f-4658-4c84-809f-e9984da40fbb\") "
Sep 11 18:18:53.444231 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.444204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1923587f-4658-4c84-809f-e9984da40fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1923587f-4658-4c84-809f-e9984da40fbb" (UID: "1923587f-4658-4c84-809f-e9984da40fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:53.446205 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.446170    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1923587f-4658-4c84-809f-e9984da40fbb" (UID: "1923587f-4658-4c84-809f-e9984da40fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:53.446317 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.446199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1923587f-4658-4c84-809f-e9984da40fbb" (UID: "1923587f-4658-4c84-809f-e9984da40fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:53.446317 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.446254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1923587f-4658-4c84-809f-e9984da40fbb-kube-api-access-frgb8" (OuterVolumeSpecName: "kube-api-access-frgb8") pod "1923587f-4658-4c84-809f-e9984da40fbb" (UID: "1923587f-4658-4c84-809f-e9984da40fbb"). InnerVolumeSpecName "kube-api-access-frgb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:53.544648 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.544612    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frgb8\" (UniqueName: \"kubernetes.io/projected/1923587f-4658-4c84-809f-e9984da40fbb-kube-api-access-frgb8\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.544648 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.544640    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-maas-api-tls\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.544648 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.544649    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1923587f-4658-4c84-809f-e9984da40fbb-openshift-service-ca\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.544648 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:53.544657    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1923587f-4658-4c84-809f-e9984da40fbb-metrics-certs\") on node \"ip-10-0-135-167.ec2.internal\" DevicePath \"\""
Sep 11 18:18:54.020460 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.020430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:18:54.021210 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.021175    2563 generic.go:356] "Generic (PLEG): container finished" podID="1923587f-4658-4c84-809f-e9984da40fbb" containerID="588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484" exitCode=0
Sep 11 18:18:54.021371 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.021224    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"
Sep 11 18:18:54.021371 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.021263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54" event={"ID":"1923587f-4658-4c84-809f-e9984da40fbb","Type":"ContainerDied","Data":"588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484"}
Sep 11 18:18:54.021371 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.021319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54" event={"ID":"1923587f-4658-4c84-809f-e9984da40fbb","Type":"ContainerDied","Data":"79ad31ecf543e90003083796a4416281792c45710972989b590f8d8be0f090d6"}
Sep 11 18:18:54.021371 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.021349    2563 scope.go:117] "RemoveContainer" containerID="588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484"
Sep 11 18:18:54.025619 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.025599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9zj98_c1805cae-72e6-4e53-b586-3625d2c70393/ovn-acl-logging/0.log"
Sep 11 18:18:54.031446 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.031427    2563 scope.go:117] "RemoveContainer" containerID="588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484"
Sep 11 18:18:54.031747 ip-10-0-135-167 kubenswrapper[2563]: E0911 18:18:54.031728    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484\": container with ID starting with 588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484 not found: ID does not exist" containerID="588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484"
Sep 11 18:18:54.031803 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.031764    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484"} err="failed to get container status \"588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484\": rpc error: code = NotFound desc = could not find container \"588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484\": container with ID starting with 588ed9ca633950c807b6551a46e1fc63cea3288a096a4b8c93c39c8c6ed21484 not found: ID does not exist"
Sep 11 18:18:54.051282 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.051207    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"]
Sep 11 18:18:54.057291 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:54.057263    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-579ffb46f9-4pv54"]
Sep 11 18:18:55.786756 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:18:55.782734    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1923587f-4658-4c84-809f-e9984da40fbb" path="/var/lib/kubelet/pods/1923587f-4658-4c84-809f-e9984da40fbb/volumes"
Sep 11 18:19:20.922554 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:20.922442    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b57799854-5m5zr_5afd6465-8760-4ac6-8591-f8d0df862014/manager/0.log"
Sep 11 18:19:21.041732 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:21.041702    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vz62r_d4f59910-f0ba-4d2d-b532-ffd0af390f9f/server/0.log"
Sep 11 18:19:21.180562 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:21.180465    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qsqsw_f2067629-41a1-4d38-8b75-5f232d8b8925/manager/0.log"
Sep 11 18:19:21.449590 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:21.449507    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86997d8dbf-fkzwm_47ec1ed1-e924-4007-9a80-f42ccf6c9350/copy-manifests/0.log"
Sep 11 18:19:21.458627 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:21.458602    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86997d8dbf-fkzwm_47ec1ed1-e924-4007-9a80-f42ccf6c9350/manager/0.log"
Sep 11 18:19:21.950186 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:21.950156    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c48469567-jc2gt_2cdf7dba-0c05-4e79-949e-bc037be20c9e/maas-api/0.log"
Sep 11 18:19:24.500681 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:24.500633    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh_4b9f65a6-44d2-4bf1-8711-6939def63e7e/istio-proxy/0.log"
Sep 11 18:19:24.633604 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:24.633568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tdmbx_38c56a04-4af2-4457-962f-357212314f6f/discovery/0.log"
Sep 11 18:19:25.016712 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:25.016684    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/1.log"
Sep 11 18:19:25.780277 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:25.780246    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-m6h2m_7fb349bc-ceac-461d-bbe7-8f1cf719a69e/main/0.log"
Sep 11 18:19:26.153370 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:26.153337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-cbg8f_380bf1fc-44af-4dfe-9f89-35155776e086/main/0.log"
Sep 11 18:19:35.950473 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:35.950437    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-94tdw_6e38817a-672b-4e85-ab7c-405f8eb555c5/global-pull-secret-syncer/0.log"
Sep 11 18:19:36.205278 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:36.205182    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wpcq7_8f2c6867-5f6f-4441-a673-1159fda5b848/konnectivity-agent/0.log"
Sep 11 18:19:36.268103 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:36.268075    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-167.ec2.internal_ab0f818a53aaf7fb073af2872df2c0e1/haproxy/0.log"
Sep 11 18:19:42.098093 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.098060    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/alertmanager/0.log"
Sep 11 18:19:42.125394 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.125366    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/config-reloader/0.log"
Sep 11 18:19:42.155222 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.155177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/kube-rbac-proxy-web/0.log"
Sep 11 18:19:42.188471 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.188440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/kube-rbac-proxy/0.log"
Sep 11 18:19:42.216965 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.216930    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/kube-rbac-proxy-metric/0.log"
Sep 11 18:19:42.249707 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.249674    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/prom-label-proxy/0.log"
Sep 11 18:19:42.277922 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.277895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c9961fae-dd3b-4e48-a2e1-ea00a01c33db/init-config-reloader/0.log"
Sep 11 18:19:42.362790 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.362690    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-r9lqm_29c1f41e-0109-4f2f-b5e1-d26237330394/kube-state-metrics/0.log"
Sep 11 18:19:42.382759 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.382698    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-r9lqm_29c1f41e-0109-4f2f-b5e1-d26237330394/kube-rbac-proxy-main/0.log"
Sep 11 18:19:42.412093 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.412062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-r9lqm_29c1f41e-0109-4f2f-b5e1-d26237330394/kube-rbac-proxy-self/0.log"
Sep 11 18:19:42.445278 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.445217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-645b98cdc6-8g2vt_d952eb63-5524-4d2b-b9c5-c40ce20966ba/metrics-server/0.log"
Sep 11 18:19:42.479424 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.479392    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-q7pnl_7c1f8b56-129e-4aac-ba8f-6933c3b84139/monitoring-plugin/0.log"
Sep 11 18:19:42.624080 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.623978    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jc6m6_9e3d2659-dc80-4b14-a6bc-c7292398abc9/node-exporter/0.log"
Sep 11 18:19:42.652943 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.652914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jc6m6_9e3d2659-dc80-4b14-a6bc-c7292398abc9/kube-rbac-proxy/0.log"
Sep 11 18:19:42.684757 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:42.684733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jc6m6_9e3d2659-dc80-4b14-a6bc-c7292398abc9/init-textfile/0.log"
Sep 11 18:19:43.174879 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:43.174849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wcflv_bea627ed-cd42-4058-9be4-f51e58deea6b/prometheus-operator/0.log"
Sep 11 18:19:43.216802 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:43.216769    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-wcflv_bea627ed-cd42-4058-9be4-f51e58deea6b/kube-rbac-proxy/0.log"
Sep 11 18:19:43.287244 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:43.287218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-kvcsq_226aa17f-6264-4f54-a35a-861ccc9bd9be/prometheus-operator-admission-webhook/0.log"
Sep 11 18:19:43.338948 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:43.338919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-764fcd64dc-j4sfl_c19e5383-0743-4d11-9b1a-fceb9969556a/telemeter-client/0.log"
Sep 11 18:19:43.375376 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:43.375261    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-764fcd64dc-j4sfl_c19e5383-0743-4d11-9b1a-fceb9969556a/reload/0.log"
Sep 11 18:19:43.409078 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:43.409053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-764fcd64dc-j4sfl_c19e5383-0743-4d11-9b1a-fceb9969556a/kube-rbac-proxy/0.log"
Sep 11 18:19:46.233938 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.233903    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"]
Sep 11 18:19:46.234343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.234299    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1923587f-4658-4c84-809f-e9984da40fbb" containerName="maas-api"
Sep 11 18:19:46.234343 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.234310    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1923587f-4658-4c84-809f-e9984da40fbb" containerName="maas-api"
Sep 11 18:19:46.234417 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.234373    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1923587f-4658-4c84-809f-e9984da40fbb" containerName="maas-api"
Sep 11 18:19:46.237720 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.237690    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.242006 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.241986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9jhl8\"/\"default-dockercfg-d7m62\""
Sep 11 18:19:46.242137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.241987    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jhl8\"/\"openshift-service-ca.crt\""
Sep 11 18:19:46.242137 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.242043    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jhl8\"/\"kube-root-ca.crt\""
Sep 11 18:19:46.251370 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.251335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"]
Sep 11 18:19:46.338959 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.338922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-sys\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.338959 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.338964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-proc\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.339230 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.338993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-podres\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.339230 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.339072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-lib-modules\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.339230 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.339112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdm95\" (UniqueName: \"kubernetes.io/projected/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-kube-api-access-rdm95\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439642 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-proc\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439642 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-podres\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-lib-modules\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdm95\" (UniqueName: \"kubernetes.io/projected/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-kube-api-access-rdm95\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-proc\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-podres\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-lib-modules\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-sys\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.439873 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.439778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-sys\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.452884 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.452850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdm95\" (UniqueName: \"kubernetes.io/projected/7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a-kube-api-access-rdm95\") pod \"perf-node-gather-daemonset-929bk\" (UID: \"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a\") " pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.547475 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.547383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:46.833905 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:46.833786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"]
Sep 11 18:19:46.836320 ip-10-0-135-167 kubenswrapper[2563]: W0911 18:19:46.836288    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f8c59d0_0c31_4e7e_bbc2_3ee592410c6a.slice/crio-c3b561e44f5f5e5ca29019596cf152c0f8e065f98a8cd2a63427a3974f4ad0ac WatchSource:0}: Error finding container c3b561e44f5f5e5ca29019596cf152c0f8e065f98a8cd2a63427a3974f4ad0ac: Status 404 returned error can't find the container with id c3b561e44f5f5e5ca29019596cf152c0f8e065f98a8cd2a63427a3974f4ad0ac
Sep 11 18:19:47.316571 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:47.316450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk" event={"ID":"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a","Type":"ContainerStarted","Data":"6766ac8ad0b22eb0eeecbf5937755905a4d970bbb639b5cd35a46337922d50df"}
Sep 11 18:19:47.316571 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:47.316528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk" event={"ID":"7f8c59d0-0c31-4e7e-bbc2-3ee592410c6a","Type":"ContainerStarted","Data":"c3b561e44f5f5e5ca29019596cf152c0f8e065f98a8cd2a63427a3974f4ad0ac"}
Sep 11 18:19:47.316979 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:47.316678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:47.341363 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:47.341306    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk" podStartSLOduration=1.341290305 podStartE2EDuration="1.341290305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:47.339478646 +0000 UTC m=+2454.102933373" watchObservedRunningTime="2026-09-11 18:19:47.341290305 +0000 UTC m=+2454.104745032"
Sep 11 18:19:48.851122 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:48.851093    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-spwdq_9c11678b-c630-4a52-b7b2-f4bb6c9f98bd/dns/0.log"
Sep 11 18:19:48.875705 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:48.875678    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-spwdq_9c11678b-c630-4a52-b7b2-f4bb6c9f98bd/kube-rbac-proxy/0.log"
Sep 11 18:19:48.927704 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:48.927672    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kthkz_7f0836d0-c705-48f4-b420-cbada6b229a9/dns-node-resolver/0.log"
Sep 11 18:19:49.587349 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:49.587317    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6mhjc_9c15f880-59bc-45ab-beff-5f7c3db77686/node-ca/0.log"
Sep 11 18:19:50.697158 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:50.697130    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh_4b9f65a6-44d2-4bf1-8711-6939def63e7e/istio-proxy/0.log"
Sep 11 18:19:51.515736 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:51.515698    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tdmbx_38c56a04-4af2-4457-962f-357212314f6f/discovery/0.log"
Sep 11 18:19:51.602477 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:51.602444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/0.log"
Sep 11 18:19:51.607578 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:51.607554    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/1.log"
Sep 11 18:19:52.651042 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:52.650999    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpfvj_447fff49-0885-43e0-9b38-df3d55ad4a92/serve-healthcheck-canary/0.log"
Sep 11 18:19:53.191327 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:53.191294    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-578g9_c530f3de-de46-4ba9-80de-5bbe9bdc6b62/kube-rbac-proxy/0.log"
Sep 11 18:19:53.213813 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:53.213777    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-578g9_c530f3de-de46-4ba9-80de-5bbe9bdc6b62/exporter/0.log"
Sep 11 18:19:53.237335 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:53.237301    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-578g9_c530f3de-de46-4ba9-80de-5bbe9bdc6b62/extractor/0.log"
Sep 11 18:19:53.333143 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:53.333112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9jhl8/perf-node-gather-daemonset-929bk"
Sep 11 18:19:56.253812 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:56.253766    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b57799854-5m5zr_5afd6465-8760-4ac6-8591-f8d0df862014/manager/0.log"
Sep 11 18:19:56.283577 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:56.283544    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vz62r_d4f59910-f0ba-4d2d-b532-ffd0af390f9f/server/0.log"
Sep 11 18:19:56.434989 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:56.434955    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-qsqsw_f2067629-41a1-4d38-8b75-5f232d8b8925/manager/0.log"
Sep 11 18:19:56.529830 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:56.529734    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh_4b9f65a6-44d2-4bf1-8711-6939def63e7e/istio-proxy/0.log"
Sep 11 18:19:56.661928 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:56.661876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86997d8dbf-fkzwm_47ec1ed1-e924-4007-9a80-f42ccf6c9350/manager/0.log"
Sep 11 18:19:56.689151 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:56.689118    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86997d8dbf-fkzwm_47ec1ed1-e924-4007-9a80-f42ccf6c9350/copy-manifests/0.log"
Sep 11 18:19:57.385371 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:57.385339    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tdmbx_38c56a04-4af2-4457-962f-357212314f6f/discovery/0.log"
Sep 11 18:19:57.435654 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:57.435619    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/0.log"
Sep 11 18:19:57.440770 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:57.440747    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/1.log"
Sep 11 18:19:58.018443 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:58.018359    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6476dc957d-595x9_ac8da5ee-1613-499f-8759-9d769c40b605/manager/0.log"
Sep 11 18:19:59.025665 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:59.025584    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-cssvh_4b9f65a6-44d2-4bf1-8711-6939def63e7e/istio-proxy/0.log"
Sep 11 18:19:59.777773 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:59.777738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-tdmbx_38c56a04-4af2-4457-962f-357212314f6f/discovery/0.log"
Sep 11 18:19:59.893385 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:59.893347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/0.log"
Sep 11 18:19:59.898668 ip-10-0-135-167 kubenswrapper[2563]: I0911 18:19:59.898625    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4grfw_769ab66a-cade-43e5-a337-c6697aaa6aca/istio-proxy/1.log"