Sep 09 20:46:37.804163 ip-10-0-134-62 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:46:37.804175 ip-10-0-134-62 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:46:37.804182 ip-10-0-134-62 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:46:37.804409 ip-10-0-134-62 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:46:48.030143 ip-10-0-134-62 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:46:48.030163 ip-10-0-134-62 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 70102beffc0d465d973b0158b926eadd --
Sep 09 20:48:31.372995 ip-10-0-134-62 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:48:31.815422 ip-10-0-134-62 kubenswrapper[2548]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:48:31.815422 ip-10-0-134-62 kubenswrapper[2548]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:48:31.815422 ip-10-0-134-62 kubenswrapper[2548]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:48:31.815422 ip-10-0-134-62 kubenswrapper[2548]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:48:31.815422 ip-10-0-134-62 kubenswrapper[2548]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:48:31.816467 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.816389    2548 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:48:31.820468 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820455    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:31.820468 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820468    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820471    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820475    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820477    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820480    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820483    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820486    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820489    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820491    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820494    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820497    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820500    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820503    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820505    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820508    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820511    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820513    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820516    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820519    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820521    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:31.820529 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820524    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820527    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820530    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820533    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820535    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820538    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820541    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820544    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820547    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820550    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820552    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820555    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820557    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820560    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820563    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820565    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820568    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820570    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820573    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:31.821027 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820575    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820578    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820580    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820583    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820586    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820588    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820591    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820593    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820596    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820598    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820602    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820606    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820609    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820612    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820615    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820618    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820620    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820623    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820625    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:31.821488 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820628    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820631    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820633    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820636    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820638    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820641    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820644    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820647    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820649    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820652    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820654    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820658    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820662    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820665    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820668    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820671    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820674    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820676    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820679    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:31.821962 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820682    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820684    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820687    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820689    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820691    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820694    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820696    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820699    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820702    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820705    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820708    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820711    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820713    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820716    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820719    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820721    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820724    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820739    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820743    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820746    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:31.822422 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.820749    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821404    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821410    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821413    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821416    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821419    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821423    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821426    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821428    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821431    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821433    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821436    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821439    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821441    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821444    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821447    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821449    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821452    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821456    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821460    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:31.822916 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821463    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821467    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821470    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821474    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821482    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821486    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821489    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821492    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821495    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821499    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821503    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821506    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821509    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821511    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821514    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821516    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821519    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821521    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821524    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:31.823512 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821527    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821529    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821532    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821534    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821537    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821539    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821542    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821544    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821547    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821549    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821553    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821557    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821561    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821563    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821566    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821569    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821572    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821574    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821577    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:31.823989 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821579    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821581    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821584    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821586    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821589    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821591    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821594    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821597    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821599    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821601    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821604    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821606    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821609    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821611    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821614    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821616    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821619    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821621    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821624    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821626    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:31.824451 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821629    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821631    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821633    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821636    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821638    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821641    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821644    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821647    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821649    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821652    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821654    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821657    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821659    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821662    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821664    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821667    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821669    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821672    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821674    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821677    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:31.824957 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821679    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.821682    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822067    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822072    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822075    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822078    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822082    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822084    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822087    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822090    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822092    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822095    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822097    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822100    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822102    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822105    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822107    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822110    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822114    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822116    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:31.825441 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822119    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822122    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822124    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822127    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822129    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822132    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822135    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822137    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822140    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822142    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822145    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822147    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822150    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822152    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822155    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822157    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822160    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822162    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822164    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822167    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:31.825958 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822170    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822173    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822175    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822178    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822181    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822183    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822186    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822188    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822191    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822195    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822201    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822204    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822206    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822209    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822212    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822214    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822217    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822220    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822224    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:31.826443 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822226    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822229    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822232    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822234    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822237    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822241    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822243    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822246    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822248    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822251    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822253    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822256    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822258    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822262    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822264    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822267    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822269    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822272    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822276    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822279    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:31.826922 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822281    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822284    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822286    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822290    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822293    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822295    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822298    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822301    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822305    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822309    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822311    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822313    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822316    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822319    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822321    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822324    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822326    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822330    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822333    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:31.827404 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822336    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822339    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.822342    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822409    2548 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822454    2548 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822463    2548 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822468    2548 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822473    2548 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822476    2548 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822480    2548 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822484    2548 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822488    2548 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822491    2548 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822494    2548 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822498    2548 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822500    2548 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822505    2548 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822507    2548 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822510    2548 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822513    2548 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822516    2548 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822519    2548 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822522    2548 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:48:31.827864 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822525    2548 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822528    2548 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822531    2548 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822535    2548 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822537    2548 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822541    2548 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822544    2548 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822547    2548 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822549    2548 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822552    2548 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822555    2548 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822558    2548 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822563    2548 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822566    2548 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822568    2548 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822571    2548 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822574    2548 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822577    2548 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822582    2548 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822585    2548 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822588    2548 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822591    2548 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822594    2548 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822597    2548 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:48:31.828465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822600    2548 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822604    2548 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822607    2548 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822610    2548 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822613    2548 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822616    2548 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822619    2548 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822621    2548 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822624    2548 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822627    2548 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822631    2548 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822633    2548 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822636    2548 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822639    2548 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822642    2548 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822645    2548 flags.go:64] FLAG: --help="false"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822648    2548 flags.go:64] FLAG: --hostname-override="ip-10-0-134-62.ec2.internal"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822651    2548 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822654    2548 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822656    2548 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822660    2548 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822663    2548 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822665    2548 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:48:31.829054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822668    2548 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822671    2548 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822673    2548 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822676    2548 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822679    2548 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822682    2548 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822685    2548 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822688    2548 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822691    2548 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822694    2548 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822698    2548 flags.go:64] FLAG: --lock-file=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822701    2548 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822704    2548 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822707    2548 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822716    2548 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822719    2548 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822722    2548 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822725    2548 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822728    2548 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822743    2548 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822746    2548 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822749    2548 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822753    2548 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822756    2548 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822760    2548 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:48:31.829658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822762    2548 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822765    2548 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822768    2548 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822771    2548 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822774    2548 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822777    2548 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822780    2548 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822786    2548 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822789    2548 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822792    2548 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822795    2548 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822799    2548 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822804    2548 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822807    2548 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822810    2548 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822812    2548 flags.go:64] FLAG: --port="10250"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822815    2548 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822819    2548 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d25620cda125766b"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822823    2548 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822825    2548 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822828    2548 flags.go:64] FLAG: --register-node="true"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822831    2548 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822839    2548 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822842    2548 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:48:31.830312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822845    2548 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822848    2548 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822851    2548 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822854    2548 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822857    2548 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822860    2548 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822862    2548 flags.go:64] FLAG: --runonce="false"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822865    2548 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822868    2548 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822871    2548 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822874    2548 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822876    2548 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822879    2548 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822882    2548 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822885    2548 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822888    2548 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822891    2548 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822894    2548 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822896    2548 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822901    2548 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822904    2548 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822907    2548 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822912    2548 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822915    2548 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822917    2548 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:48:31.830951 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822923    2548 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822926    2548 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822929    2548 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822932    2548 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822934    2548 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822937    2548 flags.go:64] FLAG: --v="2"
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822941    2548 flags.go:64] FLAG: --version="false"
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822945    2548 flags.go:64] FLAG: --vmodule=""
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822948    2548 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.822951    2548 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823057    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823061    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823064    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823067    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823070    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823072    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823076    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823078    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823081    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823083    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823086    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823089    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:31.831553 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823091    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823094    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823096    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823098    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823102    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823105    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823108    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823112    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823116    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823119    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823123    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823126    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823128    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823131    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823134    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823137    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823139    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823142    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823144    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:31.832143 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823147    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823149    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823152    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823155    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823157    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823160    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823163    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823165    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823168    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823170    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823173    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823176    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823178    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823181    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823183    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823186    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823188    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823193    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823196    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:31.832602 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823200    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823203    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823206    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823208    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823211    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823214    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823216    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823219    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823221    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823224    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823227    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823231    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823233    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823236    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823239    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823241    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823244    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823246    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823249    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823251    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:31.833102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823254    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823257    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823259    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823262    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823264    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823267    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823270    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823272    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823275    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823278    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823281    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823284    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823287    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823289    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823292    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823295    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823297    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823300    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823302    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823305    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:31.833581 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823308    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823310    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823313    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823315    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823318    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823320    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823323    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823325    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.823328    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.824016    2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831114    2548 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831128    2548 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831146    2548 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831152    2548 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831202    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:31.834102 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831206    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831210    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831213    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831217    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831221    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831224    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831227    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831230    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831232    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831235    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831238    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831241    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831244    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831247    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831249    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831252    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831254    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831257    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831259    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831262    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:31.834477 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831265    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831267    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831270    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831272    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831275    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831277    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831280    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831282    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831285    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831288    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831290    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831293    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831296    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831299    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831301    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831304    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831306    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831309    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831311    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831314    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:31.834980 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831316    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831319    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831321    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831324    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831327    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831329    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831332    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831334    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831337    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831339    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831341    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831344    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831347    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831349    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831353    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831357    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831360    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831363    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831366    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:31.835531 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831369    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831372    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831375    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831377    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831380    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831383    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831386    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831389    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831392    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831395    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831399    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831402    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831405    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831407    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831410    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831413    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831416    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831418    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831421    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:31.836019 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831424    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831426    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831429    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831431    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831434    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831436    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831439    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831441    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831444    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831447    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831449    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831452    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831454    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831457    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831460    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831463    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831465    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831468    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831470    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:31.836518 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831473    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831478    2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831578    2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831583    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831585    2548 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831589    2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831591    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831594    2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831597    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831600    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831602    2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831605    2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831607    2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831611    2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831613    2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:48:31.837005 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831616    2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831618    2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831621    2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831624    2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831626    2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831629    2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831631    2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831634    2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831636    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831638    2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831641    2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831644    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831647    2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831651    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831654    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831656    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831658    2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831661    2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831663    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:48:31.837387 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831666    2548 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831668    2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831671    2548 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831673    2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831675    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831678    2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831680    2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831683    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831685    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831688    2548 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831690    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831692    2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831695    2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831698    2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831700    2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831704    2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831707    2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831710    2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831713    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:48:31.837928 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831715    2548 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831718    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831720    2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831722    2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831725    2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831728    2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831743    2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831746    2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831749    2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831751    2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831754    2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831756    2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831759    2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831761    2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831763    2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831766    2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831768    2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831771    2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831773    2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831776    2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:48:31.838397 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831778    2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831780    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831783    2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831785    2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831788    2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831790    2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831793    2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831796    2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831798    2548 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831800    2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831803    2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831805    2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831808    2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831810    2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831813    2548 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831815    2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831818    2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831820    2548 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831823    2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831825    2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:48:31.838906 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831828    2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831830    2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831833    2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831835    2548 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831837    2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831840    2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831842    2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:31.831846    2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831851    2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.831934    2548 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.835206    2548 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.836295    2548 server.go:1019] "Starting client certificate rotation"
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.836396    2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:48:31.839392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.836429    2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:48:31.839724 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.839148    2548 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:48:31.840190 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.840173    2548 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:48:31.853064 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.853045    2548 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:48:31.856633 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.856617    2548 log.go:25] "Validated CRI v1 image API"
Sep 09 20:48:31.857493 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.857472    2548 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:48:31.861804 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.861779    2548 fs.go:135] Filesystem UUIDs: map[3fc15f53-1d4a-4bed-a216-aa3a4efd467e:/dev/nvme0n1p3 5189ae09-0d55-4dac-bd6f-e0a7c7945f0c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 09 20:48:31.861880 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.861804    2548 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:48:31.867646 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.867514    2548 manager.go:216] Machine: {Timestamp:2026-09-09 20:48:31.866236682 +0000 UTC m=+0.369066675 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094880 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fa25b8d587778caea96b4a6f663fd SystemUUID:ec2fa25b-8d58-7778-caea-96b4a6f663fd BootID:70102bef-fc0d-465d-973b-0158b926eadd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:08:74:95:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:08:74:95:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:29:d4:4c:47:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:48:31.867646 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.867628    2548 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:48:31.867818 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.867635    2548 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:48:31.867818 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.867756    2548 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:48:31.868986 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.868962    2548 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:48:31.869152 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.868988    2548 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:48:31.869903 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.869893    2548 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:48:31.869935 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.869906    2548 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:48:31.869935 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.869913    2548 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:48:31.870813 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.870802    2548 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:48:31.872815 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.872806    2548 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:48:31.877249 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877238    2548 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:48:31.877357 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877348    2548 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:48:31.877394 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877378    2548 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:48:31.877394 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877390    2548 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:48:31.877447 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877412    2548 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:48:31.877447 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877424    2548 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:48:31.877447 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877432    2548 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:48:31.877768 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877757    2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:48:31.877806 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.877775    2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:48:31.880033 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.880018    2548 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:48:31.882825 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.882809    2548 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:48:31.882909 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.882847    2548 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:48:31.883686 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883673    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:48:31.883776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883697    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:48:31.883776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883706    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:48:31.883776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883715    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:48:31.883776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883727    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:48:31.883776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883750    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:48:31.883776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883771    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:48:31.884008 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883779    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:48:31.884008 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883789    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:48:31.884008 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883797    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:48:31.884008 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883810    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:48:31.884008 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.883829    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:48:31.884700 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.884688    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:48:31.884770 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.884701    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:48:31.886533 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.886514    2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2d5np"
Sep 09 20:48:31.888379 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.888364    2548 server.go:1268] "Started kubelet"
Sep 09 20:48:31.888462 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.888417    2548 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:48:31.888531 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.888462    2548 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:48:31.888576 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.888553    2548 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:48:31.889088 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.889070    2548 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:48:31.889212 ip-10-0-134-62 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:48:31.889352 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.889272    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:48:31.889395 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.889355    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:48:31.890005 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.889991    2548 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.894238    2548 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.894440    2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2d5np"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.894078    2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-62.ec2.internal.18d3c21e2eed7698  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-62.ec2.internal,UID:ip-10-0-134-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-62.ec2.internal,},FirstTimestamp:2026-09-09 20:48:31.888340632 +0000 UTC m=+0.391170624,LastTimestamp:2026-09-09 20:48:31.888340632 +0000 UTC m=+0.391170624,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-62.ec2.internal,}"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.895338    2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.895504    2548 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896063    2548 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:48:31.896333 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.896152    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:31.896852 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896837    2548 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:48:31.896903 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896839    2548 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:48:31.896903 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896864    2548 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:48:31.896983 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896864    2548 factory.go:55] Registering systemd factory
Sep 09 20:48:31.896983 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896932    2548 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:48:31.896983 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896944    2548 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:48:31.896983 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.896951    2548 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:48:31.897162 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.897124    2548 factory.go:153] Registering CRI-O factory
Sep 09 20:48:31.897162 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.897136    2548 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:48:31.897257 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.897173    2548 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:48:31.897257 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.897187    2548 factory.go:103] Registering Raw factory
Sep 09 20:48:31.897257 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.897196    2548 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:48:31.897576 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.897561    2548 manager.go:349] Starting recovery of all containers
Sep 09 20:48:31.902724 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.902681    2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:48:31.903674 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.903645    2548 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:48:31.903792 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.903771    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:48:31.909153 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.907589    2548 manager.go:354] Recovery completed
Sep 09 20:48:31.912119 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.912105    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:31.913937 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.913919    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:31.913988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.913953    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:31.913988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.913967    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:31.914902 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.914875    2548 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:48:31.914902 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.914890    2548 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:48:31.914902 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.914904    2548 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:48:31.915052 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.914882    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:31.915592 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.915578    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:31.915650 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.915606    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:31.915650 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.915620    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:31.916903 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.916889    2548 policy_none.go:49] "None policy: Start"
Sep 09 20:48:31.916948 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.916910    2548 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:48:31.916948 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.916924    2548 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:48:31.918868 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.918852    2548 policy_none.go:47] "Start"
Sep 09 20:48:31.947367 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947353    2548 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.947384    2548 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947393    2548 server.go:87] "Starting device plugin registration server"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947666    2548 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947678    2548 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947841    2548 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947907    2548 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.947915    2548 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.948313    2548 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.948351    2548 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.961534    2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.961575    2548 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.961594    2548 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:31.961625    2548 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:48:31.974002 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:31.963774    2548 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:32.048179 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.048150    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:32.048986 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.048971    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:32.049081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.048997    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:32.049081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.049008    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:32.049081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.049033    2548 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.057433 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.057419    2548 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.057478 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.057439    2548 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-62.ec2.internal\": node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.062576 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.062561    2548 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"]
Sep 09 20:48:32.062647 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.062611    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:32.063317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.063298    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:32.063407 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.063324    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:32.063407 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.063339    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:32.065593 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.065539    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:32.065689 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.065675    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.065752 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.065702    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:32.066232 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.066218    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:32.066317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.066237    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:32.066317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.066246    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:32.066317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.066275    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:32.066317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.066299    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:32.066317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.066313    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:32.068455 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.068439    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.068535 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.068467    2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:48:32.069089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.069073    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:48:32.069185 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.069103    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:48:32.069185 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.069115    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:48:32.074814 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.074796    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.090268 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.090253    2548 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-62.ec2.internal\" not found" node="ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.093669 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.093651    2548 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-62.ec2.internal\" not found" node="ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.097085 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.097065    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c4618f261bc87b1a1fa90e96a30f818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal\" (UID: \"4c4618f261bc87b1a1fa90e96a30f818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.097142 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.097092    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c4618f261bc87b1a1fa90e96a30f818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal\" (UID: \"4c4618f261bc87b1a1fa90e96a30f818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.097142 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.097108    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d6cefe1924478617631a1883cb3324d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-62.ec2.internal\" (UID: \"6d6cefe1924478617631a1883cb3324d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.175228 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.175208    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.197285 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.197262    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c4618f261bc87b1a1fa90e96a30f818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal\" (UID: \"4c4618f261bc87b1a1fa90e96a30f818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.197363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.197303    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c4618f261bc87b1a1fa90e96a30f818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal\" (UID: \"4c4618f261bc87b1a1fa90e96a30f818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.197363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.197309    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c4618f261bc87b1a1fa90e96a30f818-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal\" (UID: \"4c4618f261bc87b1a1fa90e96a30f818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.197363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.197346    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c4618f261bc87b1a1fa90e96a30f818-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal\" (UID: \"4c4618f261bc87b1a1fa90e96a30f818\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.197363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.197352    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d6cefe1924478617631a1883cb3324d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-62.ec2.internal\" (UID: \"6d6cefe1924478617631a1883cb3324d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.197487 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.197383    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d6cefe1924478617631a1883cb3324d-config\") pod \"kube-apiserver-proxy-ip-10-0-134-62.ec2.internal\" (UID: \"6d6cefe1924478617631a1883cb3324d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.276009 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.275990    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.376577 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.376519    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.393817 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.393798    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.397484 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.397469    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:32.476789 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.476768    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.577204 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.577185    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.677964 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.677672    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.778089 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.778056    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.838036 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.838005    2548 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:48:32.838516 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.838197    2548 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:48:32.878702 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:32.878657    2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-62.ec2.internal\" not found"
Sep 09 20:48:32.900722 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.900671    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:43:32 +0000 UTC" deadline="2028-04-29 01:07:30.991798071 +0000 UTC"
Sep 09 20:48:32.900722 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.900720    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14332h18m58.091083136s"
Sep 09 20:48:32.963293 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.963267    2548 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:32.996623 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:32.996580    2548 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:33.007814 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.007696    2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:48:33.009941 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.009918    2548 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal"
Sep 09 20:48:33.016776 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.016759    2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:48:33.019772 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.019755    2548 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:33.090792 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:33.090757    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6cefe1924478617631a1883cb3324d.slice/crio-e95823d1743100f8d9b2465db7f29c279fe9ae32e71a8081f18b851e469feace WatchSource:0}: Error finding container e95823d1743100f8d9b2465db7f29c279fe9ae32e71a8081f18b851e469feace: Status 404 returned error can't find the container with id e95823d1743100f8d9b2465db7f29c279fe9ae32e71a8081f18b851e469feace
Sep 09 20:48:33.093929 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:33.093900    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4618f261bc87b1a1fa90e96a30f818.slice/crio-50f05ecd56bf4dbe7cebb1874fcd7a80c49f4d8e36ad30d277e9d336d2b16365 WatchSource:0}: Error finding container 50f05ecd56bf4dbe7cebb1874fcd7a80c49f4d8e36ad30d277e9d336d2b16365: Status 404 returned error can't find the container with id 50f05ecd56bf4dbe7cebb1874fcd7a80c49f4d8e36ad30d277e9d336d2b16365
Sep 09 20:48:33.096172 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.096149    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:48:33.383435 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.383341    2548 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:33.878943 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.878860    2548 apiserver.go:52] "Watching apiserver"
Sep 09 20:48:33.888633 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.885659    2548 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:48:33.888633 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.886615    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5","openshift-cluster-node-tuning-operator/tuned-4smm6","openshift-dns/node-resolver-dkcxd","openshift-image-registry/node-ca-nsh2j","openshift-multus/multus-kl2mm","openshift-multus/network-metrics-daemon-5hfr5","openshift-network-diagnostics/network-check-target-dx5nv","kube-system/global-pull-secret-syncer-78qgz","kube-system/konnectivity-agent-qvj9k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-g45br","openshift-network-operator/iptables-alerter-c48zl","openshift-ovn-kubernetes/ovnkube-node-rsp5l"]
Sep 09 20:48:33.890457 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.890430    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:33.890702 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:33.890675    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:33.892765 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.892742    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.895181    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.895232    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.895403    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.895466    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xzfr2\""
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.895471    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.897235    2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.897669    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.898119    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.898163    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.898326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.898163    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w2f6z\""
Sep 09 20:48:33.899811 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.899781    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.900104 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.900082    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.900600 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.900268    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:33.900600 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.900296    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7bhcb\""
Sep 09 20:48:33.902711 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.902142    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:48:33.902711 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.902279    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.902711 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.902435    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zbbcc\""
Sep 09 20:48:33.902711 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.902570    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.902711 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.902696    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:33.904879 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.904633    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:33.904879 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:33.904696    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:33.905720 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.905173    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qn5k6\""
Sep 09 20:48:33.905720 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.905194    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.905720 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.905374    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.905720 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.905467    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:48:33.905720 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.905552    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:48:33.907168 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.907148    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908005    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbbm\" (UniqueName: \"kubernetes.io/projected/870cf306-82d5-4437-9477-5c5d924506e8-kube-api-access-5lbbm\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908042    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysctl-conf\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908068    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-systemd\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908095    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-host\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908123    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42swb\" (UniqueName: \"kubernetes.io/projected/b0876fd3-3875-465d-9363-cc4d33ad1721-kube-api-access-42swb\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908150    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bklm\" (UniqueName: \"kubernetes.io/projected/f9f47960-5991-4325-8567-92e59aa5ac8f-kube-api-access-2bklm\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908174    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysconfig\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908199    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-lib-modules\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908222    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-modprobe-d\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.908772 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908262    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-kubernetes\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908934    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-run\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.908975    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-sys\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909005    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-var-lib-kubelet\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909049    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-tuned\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909087    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc7b235-2a09-46a5-9165-39e41d98d840-tmp-dir\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909114    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7znxk\" (UniqueName: \"kubernetes.io/projected/9dc7b235-2a09-46a5-9165-39e41d98d840-kube-api-access-7znxk\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909139    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909164    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9f47960-5991-4325-8567-92e59aa5ac8f-iptables-alerter-script\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909200    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f47960-5991-4325-8567-92e59aa5ac8f-host-slash\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:33.909236 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909235    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysctl-d\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909804 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909274    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0876fd3-3875-465d-9363-cc4d33ad1721-tmp\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:33.909804 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909301    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9dc7b235-2a09-46a5-9165-39e41d98d840-hosts-file\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:33.909804 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909436    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:48:33.909804 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909476    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.910010 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.909813    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:33.910715 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.910264    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rjpjf\""
Sep 09 20:48:33.910715 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.910330    2548 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:48:33.910715 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.910339    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:48:33.910715 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.910411    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.910715 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.910549    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:48:33.911124 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.911104    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:48:33.911933 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.911912    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:48:33.912926 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.912273    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6jhxf\""
Sep 09 20:48:33.912926 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.912334    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:48:33.912926 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.912537    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:33.912926 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.912580    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:48:33.914917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.914505    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:48:33.914917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.914732    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sl4rt\""
Sep 09 20:48:33.914917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.914777    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:33.914917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.914788    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:48:33.914917 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:33.914843    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:33.917827 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.917276    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:33.919198 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.919178    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:48:33.919531 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.919340    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhjrn\""
Sep 09 20:48:33.919531 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.919407    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:48:33.968414 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.968371    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal" event={"ID":"4c4618f261bc87b1a1fa90e96a30f818","Type":"ContainerStarted","Data":"50f05ecd56bf4dbe7cebb1874fcd7a80c49f4d8e36ad30d277e9d336d2b16365"}
Sep 09 20:48:33.971829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.971788    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal" event={"ID":"6d6cefe1924478617631a1883cb3324d","Type":"ContainerStarted","Data":"e95823d1743100f8d9b2465db7f29c279fe9ae32e71a8081f18b851e469feace"}
Sep 09 20:48:33.975285 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.975266    2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-78zj7"
Sep 09 20:48:33.986708 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.986687    2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-78zj7"
Sep 09 20:48:33.998499 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:33.998480    2548 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:48:34.010525 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010486    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-ovnkube-config\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.010525 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010523    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99c43972-e7f2-4535-ae3a-205b5fded1e9-agent-certs\") pod \"konnectivity-agent-qvj9k\" (UID: \"99c43972-e7f2-4535-ae3a-205b5fded1e9\") " pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.010704 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010553    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bklm\" (UniqueName: \"kubernetes.io/projected/f9f47960-5991-4325-8567-92e59aa5ac8f-kube-api-access-2bklm\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.010704 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010578    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-lib-modules\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.010704 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010640    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-modprobe-d\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.010704 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010665    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-env-overrides\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.010704 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010693    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d567137-490a-4219-8692-a154fda3cb54-ovn-node-metrics-cert\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010720    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010746    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f47960-5991-4325-8567-92e59aa5ac8f-host-slash\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010771    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6619b496-637a-4621-a857-75c2fc6c97be-serviceca\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010797    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-kubelet\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010838    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-ovnkube-script-lib\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010862    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-device-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010889    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssrjx\" (UniqueName: \"kubernetes.io/projected/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-kube-api-access-ssrjx\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010913    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-cnibin\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010939    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhx6\" (UniqueName: \"kubernetes.io/projected/7cb2c318-c35f-4b72-a32c-7d0371d26552-kube-api-access-rvhx6\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010965    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-etc-kubernetes\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.010994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.010990    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-ovn\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011015    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-cni-netd\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011042    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-socket-dir-parent\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011065    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-netns\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011091    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-cni-multus\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011125    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-cni-bin\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011152    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-kubernetes\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011186    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-tuned\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011212    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-cni-binary-copy\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011241    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7znxk\" (UniqueName: \"kubernetes.io/projected/9dc7b235-2a09-46a5-9165-39e41d98d840-kube-api-access-7znxk\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011266    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-registration-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011292    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-daemon-config\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011317    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-conf-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011344    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9dc7b235-2a09-46a5-9165-39e41d98d840-hosts-file\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011370    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysctl-conf\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011395    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-host\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011420    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42swb\" (UniqueName: \"kubernetes.io/projected/b0876fd3-3875-465d-9363-cc4d33ad1721-kube-api-access-42swb\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.011542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011452    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6619b496-637a-4621-a857-75c2fc6c97be-host\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011480    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-k8s-cni-cncf-io\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011505    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-run-netns\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011546    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-socket-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011571    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011597    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-cnibin\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011621    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-hostroot\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011663    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/813abdcc-2cfe-4ba7-95d4-42fce7903b66-kubelet-config\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011689    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-systemd-units\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011714    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-etc-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011742    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-run\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011767    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-sys\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011801    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011848    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-system-cni-dir\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011872    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trf2t\" (UniqueName: \"kubernetes.io/projected/6619b496-637a-4621-a857-75c2fc6c97be-kube-api-access-trf2t\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011895    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-cni-binary-copy\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.014466 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011919    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011944    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5vh\" (UniqueName: \"kubernetes.io/projected/8d567137-490a-4219-8692-a154fda3cb54-kube-api-access-sq5vh\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011972    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc7b235-2a09-46a5-9165-39e41d98d840-tmp-dir\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011973    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-kubernetes\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.011999    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9f47960-5991-4325-8567-92e59aa5ac8f-iptables-alerter-script\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012027    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0876fd3-3875-465d-9363-cc4d33ad1721-tmp\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012050    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-system-cni-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012075    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-cni-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012099    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-cni-bin\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012127    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlrl\" (UniqueName: \"kubernetes.io/projected/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-kube-api-access-4mlrl\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012152    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-node-log\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012180    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012207    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012232    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-multus-certs\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012269    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-systemd\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012297    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysconfig\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012321    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-sys-fs\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.015262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012345    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012371    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-lib-modules\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012488    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9dc7b235-2a09-46a5-9165-39e41d98d840-hosts-file\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012595    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-run\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012374    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-kubelet\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012643    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-host\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012646    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-modprobe-d\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012709    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysconfig\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012715    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-sys\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012772    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012802    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99c43972-e7f2-4535-ae3a-205b5fded1e9-konnectivity-ca\") pod \"konnectivity-agent-qvj9k\" (UID: \"99c43972-e7f2-4535-ae3a-205b5fded1e9\") " pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.012829    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012853    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-var-lib-kubelet\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012882    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/813abdcc-2cfe-4ba7-95d4-42fce7903b66-dbus\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.012916    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:34.512894721 +0000 UTC m=+3.015724712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012943    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-var-lib-kubelet\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012951    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysctl-d\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016050 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012960    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc7b235-2a09-46a5-9165-39e41d98d840-tmp-dir\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012988    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f47960-5991-4325-8567-92e59aa5ac8f-host-slash\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.012986    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013025    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013062    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-slash\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013093    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbbm\" (UniqueName: \"kubernetes.io/projected/870cf306-82d5-4437-9477-5c5d924506e8-kube-api-access-5lbbm\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013100    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9f47960-5991-4325-8567-92e59aa5ac8f-iptables-alerter-script\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013120    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-systemd\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013268    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-systemd\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013290    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-var-lib-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013328    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013351    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-log-socket\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013385    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysctl-conf\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.013887    2548 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:48:34.016946 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.014271    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-sysctl-d\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.018846 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.018799    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0876fd3-3875-465d-9363-cc4d33ad1721-etc-tuned\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.019101 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.019068    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0876fd3-3875-465d-9363-cc4d33ad1721-tmp\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.025561 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.025417    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bklm\" (UniqueName: \"kubernetes.io/projected/f9f47960-5991-4325-8567-92e59aa5ac8f-kube-api-access-2bklm\") pod \"iptables-alerter-c48zl\" (UID: \"f9f47960-5991-4325-8567-92e59aa5ac8f\") " pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.026404 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.026381    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7znxk\" (UniqueName: \"kubernetes.io/projected/9dc7b235-2a09-46a5-9165-39e41d98d840-kube-api-access-7znxk\") pod \"node-resolver-dkcxd\" (UID: \"9dc7b235-2a09-46a5-9165-39e41d98d840\") " pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.027114 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.027097    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42swb\" (UniqueName: \"kubernetes.io/projected/b0876fd3-3875-465d-9363-cc4d33ad1721-kube-api-access-42swb\") pod \"tuned-4smm6\" (UID: \"b0876fd3-3875-465d-9363-cc4d33ad1721\") " pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.032806 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.032721    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbbm\" (UniqueName: \"kubernetes.io/projected/870cf306-82d5-4437-9477-5c5d924506e8-kube-api-access-5lbbm\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:34.114281 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114236    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-var-lib-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114281 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114264    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114281 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114280    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-log-socket\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114295    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-ovnkube-config\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114342    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99c43972-e7f2-4535-ae3a-205b5fded1e9-agent-certs\") pod \"konnectivity-agent-qvj9k\" (UID: \"99c43972-e7f2-4535-ae3a-205b5fded1e9\") " pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114366    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-var-lib-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114389    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-env-overrides\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114424    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114428    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d567137-490a-4219-8692-a154fda3cb54-ovn-node-metrics-cert\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114475    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6619b496-637a-4621-a857-75c2fc6c97be-serviceca\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114501    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-kubelet\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.114652 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.114544    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-kubelet\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115041 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115012    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6619b496-637a-4621-a857-75c2fc6c97be-serviceca\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.115123 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115089    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-ovnkube-config\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115196 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115143    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-ovnkube-script-lib\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115196 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115163    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-device-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.115196 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115187    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssrjx\" (UniqueName: \"kubernetes.io/projected/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-kube-api-access-ssrjx\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.115351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115214    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-cnibin\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.115351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115242    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhx6\" (UniqueName: \"kubernetes.io/projected/7cb2c318-c35f-4b72-a32c-7d0371d26552-kube-api-access-rvhx6\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.115351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115268    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-etc-kubernetes\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115293    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-ovn\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115318    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-cni-netd\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115347    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-socket-dir-parent\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115373    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-netns\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115400    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-cni-multus\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115424    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-cni-bin\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115452    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-cni-binary-copy\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115480    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-registration-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115506    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-daemon-config\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115532    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-conf-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115563    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6619b496-637a-4621-a857-75c2fc6c97be-host\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115588    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-k8s-cni-cncf-io\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115615    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-run-netns\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115639    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-env-overrides\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.115657 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115651    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-socket-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115676    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115700    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-cnibin\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115725    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-hostroot\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115750    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/813abdcc-2cfe-4ba7-95d4-42fce7903b66-kubelet-config\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115771    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-systemd-units\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115787    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-etc-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115805    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115835    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-system-cni-dir\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115850    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trf2t\" (UniqueName: \"kubernetes.io/projected/6619b496-637a-4621-a857-75c2fc6c97be-kube-api-access-trf2t\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115866    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-cni-binary-copy\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115882    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115898    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5vh\" (UniqueName: \"kubernetes.io/projected/8d567137-490a-4219-8692-a154fda3cb54-kube-api-access-sq5vh\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115919    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-system-cni-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115935    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-cni-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115952    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-cni-bin\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115970    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlrl\" (UniqueName: \"kubernetes.io/projected/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-kube-api-access-4mlrl\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.116286 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.115991    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-node-log\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116008    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116024    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116039    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-multus-certs\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116055    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-systemd\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116072    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-sys-fs\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116088    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116105    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-kubelet\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116120    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116129    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8d567137-490a-4219-8692-a154fda3cb54-ovnkube-script-lib\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116292    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-cni-binary-copy\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116388    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-etc-kubernetes\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116434    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-ovn\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116473    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-cni-netd\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116515    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-daemon-config\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116525    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-socket-dir-parent\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116556    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-conf-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117150 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116569    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-netns\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116583    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6619b496-637a-4621-a857-75c2fc6c97be-host\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116608    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-k8s-cni-cncf-io\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116610    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-cni-multus\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116638    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-run-netns\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116655    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-cni-bin\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116700    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-socket-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117188    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-device-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116671    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99c43972-e7f2-4535-ae3a-205b5fded1e9-konnectivity-ca\") pod \"konnectivity-agent-qvj9k\" (UID: \"99c43972-e7f2-4535-ae3a-205b5fded1e9\") " pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.116137    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99c43972-e7f2-4535-ae3a-205b5fded1e9-konnectivity-ca\") pod \"konnectivity-agent-qvj9k\" (UID: \"99c43972-e7f2-4535-ae3a-205b5fded1e9\") " pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117291    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/813abdcc-2cfe-4ba7-95d4-42fce7903b66-dbus\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117333    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117374    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117406    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-slash\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117513    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-slash\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117531    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-system-cni-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117572    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-cnibin\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.117989 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117621    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/813abdcc-2cfe-4ba7-95d4-42fce7903b66-dbus\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117659    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117661    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117703    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-log-socket\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117818    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-cnibin\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117837    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d567137-490a-4219-8692-a154fda3cb54-ovn-node-metrics-cert\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117877    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-hostroot\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.117953    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/813abdcc-2cfe-4ba7-95d4-42fce7903b66-kubelet-config\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118210    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118541    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cb2c318-c35f-4b72-a32c-7d0371d26552-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118660    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-multus-cni-dir\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118726    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99c43972-e7f2-4535-ae3a-205b5fded1e9-agent-certs\") pod \"konnectivity-agent-qvj9k\" (UID: \"99c43972-e7f2-4535-ae3a-205b5fded1e9\") " pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118732    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-cni-binary-copy\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.118829 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118766    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-var-lib-cni-bin\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118912    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-sys-fs\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.118978    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119240    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-run-systemd\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119220    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.119284    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119316    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-node-log\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119331    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-kubelet\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119352    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-host-run-ovn-kubernetes\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119382    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-host-run-multus-certs\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.119424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119406    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cb2c318-c35f-4b72-a32c-7d0371d26552-system-cni-dir\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.119891 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119440    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-etc-openvswitch\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119891 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119495    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8d567137-490a-4219-8692-a154fda3cb54-systemd-units\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.119891 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.119006    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-registration-dir\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.119891 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.119682    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:34.619638084 +0000 UTC m=+3.122468084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:34.124590 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.124561    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhx6\" (UniqueName: \"kubernetes.io/projected/7cb2c318-c35f-4b72-a32c-7d0371d26552-kube-api-access-rvhx6\") pod \"multus-additional-cni-plugins-g45br\" (UID: \"7cb2c318-c35f-4b72-a32c-7d0371d26552\") " pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.125912 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.125868    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssrjx\" (UniqueName: \"kubernetes.io/projected/dd54457d-84f0-4221-b6d7-a92c38c5f1ae-kube-api-access-ssrjx\") pod \"aws-ebs-csi-driver-node-lk5m5\" (UID: \"dd54457d-84f0-4221-b6d7-a92c38c5f1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.126974 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.126773    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:34.126974 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.126799    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:34.126974 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.126811    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:34.126974 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.126881    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:34.626862478 +0000 UTC m=+3.129692464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:34.129076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.129019    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlrl\" (UniqueName: \"kubernetes.io/projected/fe6deb28-331a-4254-ad7a-fbb6bcabf1ee-kube-api-access-4mlrl\") pod \"multus-kl2mm\" (UID: \"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee\") " pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.129076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.129028    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trf2t\" (UniqueName: \"kubernetes.io/projected/6619b496-637a-4621-a857-75c2fc6c97be-kube-api-access-trf2t\") pod \"node-ca-nsh2j\" (UID: \"6619b496-637a-4621-a857-75c2fc6c97be\") " pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.130183 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.130141    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5vh\" (UniqueName: \"kubernetes.io/projected/8d567137-490a-4219-8692-a154fda3cb54-kube-api-access-sq5vh\") pod \"ovnkube-node-rsp5l\" (UID: \"8d567137-490a-4219-8692-a154fda3cb54\") " pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.209430 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.209393    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c48zl"
Sep 09 20:48:34.228980 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.228958    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nsh2j"
Sep 09 20:48:34.241932 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.241910    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4smm6"
Sep 09 20:48:34.247754 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.247722    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dkcxd"
Sep 09 20:48:34.256439 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.256423    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kl2mm"
Sep 09 20:48:34.264226 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.264204    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:34.271052 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.271031    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5"
Sep 09 20:48:34.281866 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.281848    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:34.286543 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.286525    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g45br"
Sep 09 20:48:34.304775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.304753    2548 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:48:34.521006 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.520929    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:34.521169 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.521073    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:34.521169 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.521147    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:35.521125419 +0000 UTC m=+4.023955404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:34.621394 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.621360    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:34.621573 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.621473    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:34.621573 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.621546    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:35.62152674 +0000 UTC m=+4.124356731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:34.722068 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.722032    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:34.722230 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.722202    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:34.722230 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.722226    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:34.722311 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.722235    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:34.722311 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:34.722290    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:35.722274068 +0000 UTC m=+4.225104049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:34.990437 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.990345    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:43:34 +0000 UTC" deadline="2028-02-20 12:13:14.405978979 +0000 UTC"
Sep 09 20:48:34.990437 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:34.990383    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12687h24m39.415599636s"
Sep 09 20:48:35.185074 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:35.184866    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd54457d_84f0_4221_b6d7_a92c38c5f1ae.slice/crio-4bb25b857b093451da2408e3ce8961963d5e368bb734ed0085cc893e113f006b WatchSource:0}: Error finding container 4bb25b857b093451da2408e3ce8961963d5e368bb734ed0085cc893e113f006b: Status 404 returned error can't find the container with id 4bb25b857b093451da2408e3ce8961963d5e368bb734ed0085cc893e113f006b
Sep 09 20:48:35.186573 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:35.186544    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d567137_490a_4219_8692_a154fda3cb54.slice/crio-899aeaa4c6540d03fe498ba39e50d6b665e83aa6911a30b18f437359573e8676 WatchSource:0}: Error finding container 899aeaa4c6540d03fe498ba39e50d6b665e83aa6911a30b18f437359573e8676: Status 404 returned error can't find the container with id 899aeaa4c6540d03fe498ba39e50d6b665e83aa6911a30b18f437359573e8676
Sep 09 20:48:35.188762 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:35.188639    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f47960_5991_4325_8567_92e59aa5ac8f.slice/crio-dfc12837f4fce91a594d8d0c7ff432c75cfe8623194b7ef4343e5483c210bad7 WatchSource:0}: Error finding container dfc12837f4fce91a594d8d0c7ff432c75cfe8623194b7ef4343e5483c210bad7: Status 404 returned error can't find the container with id dfc12837f4fce91a594d8d0c7ff432c75cfe8623194b7ef4343e5483c210bad7
Sep 09 20:48:35.190239 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:35.190213    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0876fd3_3875_465d_9363_cc4d33ad1721.slice/crio-8cc6ad1ead6dbc267b10e690134613c963a044c96e9227f707ceb4831270d53c WatchSource:0}: Error finding container 8cc6ad1ead6dbc267b10e690134613c963a044c96e9227f707ceb4831270d53c: Status 404 returned error can't find the container with id 8cc6ad1ead6dbc267b10e690134613c963a044c96e9227f707ceb4831270d53c
Sep 09 20:48:35.191947 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:35.191923    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6deb28_331a_4254_ad7a_fbb6bcabf1ee.slice/crio-be770cc685b7fa7e6002b826a323b7c6ba1742a554c820e19c0dfacab496ade0 WatchSource:0}: Error finding container be770cc685b7fa7e6002b826a323b7c6ba1742a554c820e19c0dfacab496ade0: Status 404 returned error can't find the container with id be770cc685b7fa7e6002b826a323b7c6ba1742a554c820e19c0dfacab496ade0
Sep 09 20:48:35.193372 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:48:35.193006    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c43972_e7f2_4535_ae3a_205b5fded1e9.slice/crio-1e00907667e61693b08c271d9735a1eeb6d014f7c152bdee4f797745c7310fbf WatchSource:0}: Error finding container 1e00907667e61693b08c271d9735a1eeb6d014f7c152bdee4f797745c7310fbf: Status 404 returned error can't find the container with id 1e00907667e61693b08c271d9735a1eeb6d014f7c152bdee4f797745c7310fbf
Sep 09 20:48:35.527937 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.527915    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:35.528076 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.528059    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:35.528129 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.528118    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:37.528098105 +0000 UTC m=+6.030928091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:35.629206 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.629175    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:35.629328 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.629274    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:35.629328 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.629323    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:37.629310553 +0000 UTC m=+6.132140533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:35.730304 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.730269    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:35.730463 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.730442    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:35.730463 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.730461    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:35.730581 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.730474    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:35.730581 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.730531    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:37.730512789 +0000 UTC m=+6.233342772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:35.963844 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.962940    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:35.963844 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.963070    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:35.963844 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.963473    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:35.963844 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.963573    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:35.963844 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.963648    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:35.963844 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:35.963733    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:35.980979 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.980618    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerStarted","Data":"316c913d223eca1ddf019ed1b6dee0775d0b808affabb0c80a667790f779c08a"}
Sep 09 20:48:35.983986 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.983957    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dkcxd" event={"ID":"9dc7b235-2a09-46a5-9165-39e41d98d840","Type":"ContainerStarted","Data":"4f0203f8ab8ec594cca9e9710be14703d792d0d2635156dcb48855d3b0f9dd5e"}
Sep 09 20:48:35.991048 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.990996    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:43:34 +0000 UTC" deadline="2028-06-19 08:16:05.682629122 +0000 UTC"
Sep 09 20:48:35.991048 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.991024    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15563h27m29.691609553s"
Sep 09 20:48:35.995438 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:35.995412    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qvj9k" event={"ID":"99c43972-e7f2-4535-ae3a-205b5fded1e9","Type":"ContainerStarted","Data":"1e00907667e61693b08c271d9735a1eeb6d014f7c152bdee4f797745c7310fbf"}
Sep 09 20:48:36.014642 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.014616    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kl2mm" event={"ID":"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee","Type":"ContainerStarted","Data":"be770cc685b7fa7e6002b826a323b7c6ba1742a554c820e19c0dfacab496ade0"}
Sep 09 20:48:36.020710 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.020659    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal" event={"ID":"6d6cefe1924478617631a1883cb3324d","Type":"ContainerStarted","Data":"79e71cc24b0163ab67d2d09c5aa2ea688ed2bcc98716466a8b7ca1175e12c9de"}
Sep 09 20:48:36.039963 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.039932    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nsh2j" event={"ID":"6619b496-637a-4621-a857-75c2fc6c97be","Type":"ContainerStarted","Data":"914f1b62404e7d5cb52df02883820d8d7099b43b8bebdaad8b620b99be21b579"}
Sep 09 20:48:36.050759 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.050732    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4smm6" event={"ID":"b0876fd3-3875-465d-9363-cc4d33ad1721","Type":"ContainerStarted","Data":"8cc6ad1ead6dbc267b10e690134613c963a044c96e9227f707ceb4831270d53c"}
Sep 09 20:48:36.060849 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.060819    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c48zl" event={"ID":"f9f47960-5991-4325-8567-92e59aa5ac8f","Type":"ContainerStarted","Data":"dfc12837f4fce91a594d8d0c7ff432c75cfe8623194b7ef4343e5483c210bad7"}
Sep 09 20:48:36.068680 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.068654    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"899aeaa4c6540d03fe498ba39e50d6b665e83aa6911a30b18f437359573e8676"}
Sep 09 20:48:36.074106 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.074082    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5" event={"ID":"dd54457d-84f0-4221-b6d7-a92c38c5f1ae","Type":"ContainerStarted","Data":"4bb25b857b093451da2408e3ce8961963d5e368bb734ed0085cc893e113f006b"}
Sep 09 20:48:36.078442 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.078417    2548 generic.go:356] "Generic (PLEG): container finished" podID="4c4618f261bc87b1a1fa90e96a30f818" containerID="a7dde4f0d95d00fae02279bdb91fa60055450534ab1dc1767c6bdb1650b093c5" exitCode=0
Sep 09 20:48:36.078544 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.078453    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal" event={"ID":"4c4618f261bc87b1a1fa90e96a30f818","Type":"ContainerDied","Data":"a7dde4f0d95d00fae02279bdb91fa60055450534ab1dc1767c6bdb1650b093c5"}
Sep 09 20:48:36.095015 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:36.094971    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-62.ec2.internal" podStartSLOduration=3.094954376 podStartE2EDuration="3.094954376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:36.040690741 +0000 UTC m=+4.543520747" watchObservedRunningTime="2026-09-09 20:48:36.094954376 +0000 UTC m=+4.597784382"
Sep 09 20:48:37.094171 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.094089    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal" event={"ID":"4c4618f261bc87b1a1fa90e96a30f818","Type":"ContainerStarted","Data":"a59ae659a83aad7551973bb19eb401176a30580c3be75fb330b083aa991713fb"}
Sep 09 20:48:37.548507 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.548422    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:37.548671 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.548589    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:37.548671 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.548650    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:41.548630019 +0000 UTC m=+10.051460005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:37.648944 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.648904    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:37.649131 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.649087    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:37.649197 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.649144    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:41.649126416 +0000 UTC m=+10.151956402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:37.749468 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.749393    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:37.749646 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.749595    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:37.749646 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.749620    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:37.749646 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.749636    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:37.749839 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.749697    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:41.749678644 +0000 UTC m=+10.252508630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:37.966027 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.965064    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:37.966027 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.965213    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:37.966027 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.965608    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:37.966027 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.965701    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:37.966027 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:37.965757    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:37.966027 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:37.965844    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:39.963350 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:39.962797    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:39.963350 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:39.962945    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:39.963350 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:39.963342    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:39.963945 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:39.963427    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:39.963945 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:39.963492    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:39.963945 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:39.963555    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:41.585838 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:41.585275    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:41.585838 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.585419    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:41.585838 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.585487    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:49.585467517 +0000 UTC m=+18.088297518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:41.686244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:41.685673    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:41.686244 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.685869    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:41.686244 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.685930    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:49.685913146 +0000 UTC m=+18.188743131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:41.786964 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:41.786927    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:41.787138 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.787104    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:41.787211 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.787139    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:41.787211 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.787152    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:41.787311 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.787210    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:48:49.787191612 +0000 UTC m=+18.290021593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:41.964167 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:41.964085    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:41.964314 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.964272    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:41.965609 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:41.964604    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:41.965609 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.964703    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:41.965609 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:41.964774    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:41.965609 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:41.965566    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:43.962877 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:43.962845    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:43.963449 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:43.962992    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:43.963449 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:43.963344    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:43.963449 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:43.963436    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:43.963608 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:43.963490    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:43.963608 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:43.963554    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:45.962683 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:45.962433    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:45.963117 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:45.962444    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:45.963117 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:45.962724    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:45.963117 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:45.962474    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:45.963117 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:45.962822    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:45.963117 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:45.962878    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:47.962431 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:47.962382    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:47.962865 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:47.962395    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:47.962865 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:47.962503    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:47.962865 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:47.962526    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:47.962865 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:47.962632    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:47.962865 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:47.962750    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:49.647577 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:49.647483    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:49.648083 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.647657    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:49.648083 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.647743    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:05.647707246 +0000 UTC m=+34.150537226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:48:49.748299 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:49.748264    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:49.748467 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.748411    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:49.748533 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.748477    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:05.748457247 +0000 UTC m=+34.251287226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:48:49.849441 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:49.849409    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:49.849612 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.849589    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:48:49.849612 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.849612    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:48:49.849763 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.849625    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:49.849763 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.849685    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:05.849665879 +0000 UTC m=+34.352495874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:48:49.961868 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:49.961791    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:49.962027 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:49.961791    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:49.962027 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.961918    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:49.962027 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:49.961791    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:49.962027 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.962008    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:49.962189 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:49.962108    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:51.962479 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:51.962446    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:51.962944 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:51.962557    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:51.962944 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:51.962602    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:51.962944 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:51.962706    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:51.962944 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:51.962604    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:51.962944 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:51.962829    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:53.135044 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.135007    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerStarted","Data":"7d9f6a3efca3559852b747023a8f6fa872263455681ac30c1525988cee1c6ad6"}
Sep 09 20:48:53.141660 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.141450    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qvj9k" event={"ID":"99c43972-e7f2-4535-ae3a-205b5fded1e9","Type":"ContainerStarted","Data":"782fd5bcbc03ef2ed6a4938ca43abb43b84b583b0a39a1dca9047490c0166776"}
Sep 09 20:48:53.144699 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.144668    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kl2mm" event={"ID":"fe6deb28-331a-4254-ad7a-fbb6bcabf1ee","Type":"ContainerStarted","Data":"88b1102638e199c2c2d0bcb6edd604f072fcd7237861e58449bc58443eae0a6f"}
Sep 09 20:48:53.146638 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.146612    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4smm6" event={"ID":"b0876fd3-3875-465d-9363-cc4d33ad1721","Type":"ContainerStarted","Data":"7bcab0a78260e1f288617093555f4e7152932e7cb029a3bf711ac0028c86ce2c"}
Sep 09 20:48:53.148584 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.148564    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5" event={"ID":"dd54457d-84f0-4221-b6d7-a92c38c5f1ae","Type":"ContainerStarted","Data":"c91dcb4711b6448f95417f7ac2d286b3360933c7559eefc4d9acc854f83b9974"}
Sep 09 20:48:53.160168 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.160106    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-62.ec2.internal" podStartSLOduration=20.160087806 podStartE2EDuration="20.160087806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:37.111915946 +0000 UTC m=+5.614745956" watchObservedRunningTime="2026-09-09 20:48:53.160087806 +0000 UTC m=+21.662917809"
Sep 09 20:48:53.174584 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.174482    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kl2mm" podStartSLOduration=3.414522117 podStartE2EDuration="21.174465601s" totalImagesPullingTime="17.759943484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:53.174210243 +0000 UTC m=+21.677040246" watchObservedRunningTime="2026-09-09 20:48:53.174465601 +0000 UTC m=+21.677295602"
Sep 09 20:48:53.190814 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.190768    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4smm6" podStartSLOduration=3.430203074 podStartE2EDuration="21.190747583s" totalImagesPullingTime="17.760544509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:53.190520535 +0000 UTC m=+21.693350539" watchObservedRunningTime="2026-09-09 20:48:53.190747583 +0000 UTC m=+21.693577577"
Sep 09 20:48:53.321326 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.321296    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:53.321871 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.321851    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:53.337749 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.337679    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qvj9k" podStartSLOduration=7.282082993 podStartE2EDuration="21.337656898s" totalImagesPullingTime="14.055573905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:53.203615338 +0000 UTC m=+21.706445340" watchObservedRunningTime="2026-09-09 20:48:53.337656898 +0000 UTC m=+21.840486899"
Sep 09 20:48:53.962104 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.962068    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:53.962279 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.962068    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:53.962279 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:53.962204    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:53.962358 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:53.962310    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:53.962358 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:53.962345    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:53.962441 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:53.962424    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:54.153797 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.153762    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"405b66cd3e66d0324ece215f1d6e51ce41e95b7c36e0537b286fcd610b046246"}
Sep 09 20:48:54.154557 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.153848    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"9aa1d9721d0f413289968997923cebe1ef4bc0954233a0f6bf7ca61d7f31f7e1"}
Sep 09 20:48:54.154557 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.153886    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"be2ba15c18a032e85502e0c55619df5d7826b678365a29aa383aebcdd9a33ccc"}
Sep 09 20:48:54.154557 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.153917    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"bc05ae6ecee7e84f11212d9c474704b17d22f91630a0dfaa0252ca21019c35a2"}
Sep 09 20:48:54.154557 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.153958    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"338ae4545fabe158eea691eaceaff28e6ebbf846f06283381bdb41b177a65ad5"}
Sep 09 20:48:54.154557 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.154001    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"fb7e377ae6931075f5c1e8611626f99cd7a8233b86d32eabd5ab3765ecbc8341"}
Sep 09 20:48:54.155149 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.155120    2548 generic.go:356] "Generic (PLEG): container finished" podID="7cb2c318-c35f-4b72-a32c-7d0371d26552" containerID="7d9f6a3efca3559852b747023a8f6fa872263455681ac30c1525988cee1c6ad6" exitCode=0
Sep 09 20:48:54.155267 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.155209    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerDied","Data":"7d9f6a3efca3559852b747023a8f6fa872263455681ac30c1525988cee1c6ad6"}
Sep 09 20:48:54.156836 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.156761    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dkcxd" event={"ID":"9dc7b235-2a09-46a5-9165-39e41d98d840","Type":"ContainerStarted","Data":"d9808fe06eb5ab0459b6367da82f3e0609490778b4edfaef0e92c75f7740bb97"}
Sep 09 20:48:54.158265 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.158233    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nsh2j" event={"ID":"6619b496-637a-4621-a857-75c2fc6c97be","Type":"ContainerStarted","Data":"5ec64937174ea057ea0247a0fae9e76e0d0ed22d82056baa62850561c540ae1c"}
Sep 09 20:48:54.158890 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.158840    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:54.159167 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.159155    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qvj9k"
Sep 09 20:48:54.197921 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.197879    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nsh2j" podStartSLOduration=4.451961797 podStartE2EDuration="22.197865974s" totalImagesPullingTime="17.745904177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:54.197508556 +0000 UTC m=+22.700338557" watchObservedRunningTime="2026-09-09 20:48:54.197865974 +0000 UTC m=+22.700695975"
Sep 09 20:48:54.210279 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.210240    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dkcxd" podStartSLOduration=4.5668397370000005 podStartE2EDuration="22.210226101s" totalImagesPullingTime="17.643386364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:54.209754046 +0000 UTC m=+22.712584041" watchObservedRunningTime="2026-09-09 20:48:54.210226101 +0000 UTC m=+22.713056102"
Sep 09 20:48:54.786618 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.786400    2548 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:48:54.959049 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.958839    2548 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:48:54.78660377Z","UUID":"775f5e3a-3aa4-4609-a95c-e59350b09c5d","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:48:54.961883 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.961856    2548 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:48:54.961883 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:54.961890    2548 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:48:55.164118 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:55.164084    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c48zl" event={"ID":"f9f47960-5991-4325-8567-92e59aa5ac8f","Type":"ContainerStarted","Data":"89a70d2398c1006c0fcd3cd4dbe249c12bc51fe55b7e39432c079fb3973ee21a"}
Sep 09 20:48:55.165982 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:55.165953    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5" event={"ID":"dd54457d-84f0-4221-b6d7-a92c38c5f1ae","Type":"ContainerStarted","Data":"3df3ce6175e88d250183ecce037ad5fd8d8294001a97332ba938f37f8564b268"}
Sep 09 20:48:55.179157 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:55.179115    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c48zl" podStartSLOduration=5.504644911 podStartE2EDuration="23.179096414s" totalImagesPullingTime="17.674451503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:55.178880613 +0000 UTC m=+23.681710614" watchObservedRunningTime="2026-09-09 20:48:55.179096414 +0000 UTC m=+23.681926415"
Sep 09 20:48:55.962594 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:55.962561    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:55.962594 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:55.962580    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:55.962824 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:55.962693    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:55.962824 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:55.962725    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:55.962932 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:55.962808    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:55.962932 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:55.962892    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:56.172173 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:56.172075    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"07bf9f38661659093eae7a57e9b3f10efd911f8358ec792b63bfdcd937ca58c8"}
Sep 09 20:48:56.174551 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:56.174511    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5" event={"ID":"dd54457d-84f0-4221-b6d7-a92c38c5f1ae","Type":"ContainerStarted","Data":"28860c48074f10dc0003a8987a3478a5b01b36507ddafd4200028e081ef9a5a3"}
Sep 09 20:48:56.191703 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:56.191659    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lk5m5" podStartSLOduration=3.831409025 podStartE2EDuration="24.191645554s" totalImagesPullingTime="20.360236529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:56.191263101 +0000 UTC m=+24.694093104" watchObservedRunningTime="2026-09-09 20:48:56.191645554 +0000 UTC m=+24.694475589"
Sep 09 20:48:57.962809 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:57.962779    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:57.963230 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:57.962778    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:57.963230 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:57.962891    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:57.963230 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:57.962970    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:48:57.963230 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:57.962788    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:57.963230 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:57.963046    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:59.184837 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.184601    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" event={"ID":"8d567137-490a-4219-8692-a154fda3cb54","Type":"ContainerStarted","Data":"592a28437cc2337498b648140d5464e0b43f78331a9dce4cedabf30e713ce114"}
Sep 09 20:48:59.185344 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.184908    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:59.187690 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.187651    2548 generic.go:356] "Generic (PLEG): container finished" podID="7cb2c318-c35f-4b72-a32c-7d0371d26552" containerID="37a1f7248549de555bd067016b3d9a372ddfb5eecaefa3410ca7b20bedbfec55" exitCode=0
Sep 09 20:48:59.187849 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.187707    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerDied","Data":"37a1f7248549de555bd067016b3d9a372ddfb5eecaefa3410ca7b20bedbfec55"}
Sep 09 20:48:59.201628 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.201608    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:48:59.210414 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.210377    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l" podStartSLOduration=9.37276976 podStartE2EDuration="27.210363156s" totalImagesPullingTime="17.837593396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:59.20979789 +0000 UTC m=+27.712627890" watchObservedRunningTime="2026-09-09 20:48:59.210363156 +0000 UTC m=+27.713193182"
Sep 09 20:48:59.962352 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.962322    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:48:59.962590 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.962322    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:48:59.962590 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:48:59.962443    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:48:59.962590 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:59.962441    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:48:59.962590 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:59.962574    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:48:59.962801 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:48:59.962667    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:49:00.192164 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.191875    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:49:00.192164 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.191977    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:49:00.208758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.208716    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:49:00.376766 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.376523    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5hfr5"]
Sep 09 20:49:00.376924 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.376908    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:49:00.377063 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:00.377030    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:49:00.377254 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.377232    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dx5nv"]
Sep 09 20:49:00.377431 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.377339    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:00.377488 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:00.377433    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:49:00.387393 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.387372    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-78qgz"]
Sep 09 20:49:00.387498 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:00.387458    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:00.387564 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:00.387547    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:49:01.197791 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:01.197761    2548 generic.go:356] "Generic (PLEG): container finished" podID="7cb2c318-c35f-4b72-a32c-7d0371d26552" containerID="3816bd2ad7ae4cfc2795fd1794f7e2a2bcb094a1cde7d98411bdf9fce331bc01" exitCode=0
Sep 09 20:49:01.198246 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:01.197839    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerDied","Data":"3816bd2ad7ae4cfc2795fd1794f7e2a2bcb094a1cde7d98411bdf9fce331bc01"}
Sep 09 20:49:01.963340 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:01.963320    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:01.963434 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:01.963397    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:49:01.963502 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:01.963483    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:01.963611 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:01.963585    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:49:01.963611 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:01.963608    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:49:01.963796 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:01.963693    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:49:02.202602 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:02.202518    2548 generic.go:356] "Generic (PLEG): container finished" podID="7cb2c318-c35f-4b72-a32c-7d0371d26552" containerID="e28d2d31d928e621c9a086979a0c5fffc334bb06e1d5eefc3c40c1d4d0efdf17" exitCode=0
Sep 09 20:49:02.203055 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:02.202589    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerDied","Data":"e28d2d31d928e621c9a086979a0c5fffc334bb06e1d5eefc3c40c1d4d0efdf17"}
Sep 09 20:49:03.962186 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:03.962154    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:03.962641 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:03.962163    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:03.962641 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:03.962310    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:49:03.962641 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:03.962276    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-78qgz" podUID="813abdcc-2cfe-4ba7-95d4-42fce7903b66"
Sep 09 20:49:03.962641 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:03.962390    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx5nv" podUID="23cb1fd8-9dda-4c9b-8e99-513654529705"
Sep 09 20:49:03.962641 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:03.962464    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:49:05.659644 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.659599    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:49:05.660092 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.659772    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:49:05.660092 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.659851    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:37.659819762 +0000 UTC m=+66.162649741 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:49:05.760246 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.760165    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:05.760400 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.760346    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:49:05.760442 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.760425    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret podName:813abdcc-2cfe-4ba7-95d4-42fce7903b66 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:37.760405611 +0000 UTC m=+66.263235594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret") pod "global-pull-secret-syncer-78qgz" (UID: "813abdcc-2cfe-4ba7-95d4-42fce7903b66") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:49:05.777813 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.777783    2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-62.ec2.internal" event="NodeReady"
Sep 09 20:49:05.777987 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.777938    2548 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:49:05.820415 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.820381    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wl49r"]
Sep 09 20:49:05.849410 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.849367    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xcvt7"]
Sep 09 20:49:05.849575 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.849494    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:05.851925 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.851711    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:49:05.851925 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.851811    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zq52\""
Sep 09 20:49:05.852200 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.852167    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:49:05.861113 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.861085    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:05.861252 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.861235    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:49:05.861314 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.861258    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:49:05.861314 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.861279    2548 projected.go:196] Error preparing data for projected volume kube-api-access-thtgb for pod openshift-network-diagnostics/network-check-target-dx5nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:49:05.861397 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:05.861329    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb podName:23cb1fd8-9dda-4c9b-8e99-513654529705 nodeName:}" failed. No retries permitted until 2026-09-09 20:49:37.861315718 +0000 UTC m=+66.364145701 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-thtgb" (UniqueName: "kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb") pod "network-check-target-dx5nv" (UID: "23cb1fd8-9dda-4c9b-8e99-513654529705") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:49:05.863354 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.863333    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wl49r"]
Sep 09 20:49:05.863448 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.863380    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xcvt7"]
Sep 09 20:49:05.863487 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.863464    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:05.865629 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.865565    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:49:05.865629 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.865595    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:49:05.865810 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.865597    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b96lg\""
Sep 09 20:49:05.865810 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.865600    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:49:05.961901 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.961796    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086ea7e4-5ed3-41cc-9842-287c26d02fc7-cert\") pod \"ingress-canary-xcvt7\" (UID: \"086ea7e4-5ed3-41cc-9842-287c26d02fc7\") " pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:05.961901 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.961839    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce390400-86f1-4cf0-8155-8313e802c2da-tmp-dir\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:05.961901 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.961864    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:05.961901 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.961901    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfxf\" (UniqueName: \"kubernetes.io/projected/086ea7e4-5ed3-41cc-9842-287c26d02fc7-kube-api-access-pmfxf\") pod \"ingress-canary-xcvt7\" (UID: \"086ea7e4-5ed3-41cc-9842-287c26d02fc7\") " pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:05.962244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.961942    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2htk\" (UniqueName: \"kubernetes.io/projected/ce390400-86f1-4cf0-8155-8313e802c2da-kube-api-access-x2htk\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:05.962244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.961967    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce390400-86f1-4cf0-8155-8313e802c2da-config-volume\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:05.962244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.962014    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:49:05.962244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.962228    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:05.962412 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.962383    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:05.964653 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.964622    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6qsr\""
Sep 09 20:49:05.964822 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.964674    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:49:05.964822 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.964698    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m7n9z\""
Sep 09 20:49:05.964822 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.964721    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:49:05.964822 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.964758    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:49:05.965195 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:05.965171    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:49:06.062788 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.062696    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce390400-86f1-4cf0-8155-8313e802c2da-config-volume\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.062953 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.062798    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086ea7e4-5ed3-41cc-9842-287c26d02fc7-cert\") pod \"ingress-canary-xcvt7\" (UID: \"086ea7e4-5ed3-41cc-9842-287c26d02fc7\") " pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:06.062953 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.062820    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce390400-86f1-4cf0-8155-8313e802c2da-tmp-dir\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.062953 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.062845    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.062953 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.062901    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfxf\" (UniqueName: \"kubernetes.io/projected/086ea7e4-5ed3-41cc-9842-287c26d02fc7-kube-api-access-pmfxf\") pod \"ingress-canary-xcvt7\" (UID: \"086ea7e4-5ed3-41cc-9842-287c26d02fc7\") " pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:06.062953 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.062945    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2htk\" (UniqueName: \"kubernetes.io/projected/ce390400-86f1-4cf0-8155-8313e802c2da-kube-api-access-x2htk\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.063196 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.063167    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce390400-86f1-4cf0-8155-8313e802c2da-tmp-dir\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.063295 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:06.063276    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:06.063356 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:06.063344    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:49:06.563323775 +0000 UTC m=+35.066153756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:06.063416 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.063354    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ce390400-86f1-4cf0-8155-8313e802c2da-config-volume\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.067377 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.067347    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/086ea7e4-5ed3-41cc-9842-287c26d02fc7-cert\") pod \"ingress-canary-xcvt7\" (UID: \"086ea7e4-5ed3-41cc-9842-287c26d02fc7\") " pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:06.071124 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.071098    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2htk\" (UniqueName: \"kubernetes.io/projected/ce390400-86f1-4cf0-8155-8313e802c2da-kube-api-access-x2htk\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.074358 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.074337    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfxf\" (UniqueName: \"kubernetes.io/projected/086ea7e4-5ed3-41cc-9842-287c26d02fc7-kube-api-access-pmfxf\") pod \"ingress-canary-xcvt7\" (UID: \"086ea7e4-5ed3-41cc-9842-287c26d02fc7\") " pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:06.173386 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.173348    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xcvt7"
Sep 09 20:49:06.513129 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.512872    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xcvt7"]
Sep 09 20:49:06.516994 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:49:06.516966    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086ea7e4_5ed3_41cc_9842_287c26d02fc7.slice/crio-c8fb41ecca9dbd85abd3e625d453690cd2e2d25ccf8ceaca4ae5e67f6b3249c0 WatchSource:0}: Error finding container c8fb41ecca9dbd85abd3e625d453690cd2e2d25ccf8ceaca4ae5e67f6b3249c0: Status 404 returned error can't find the container with id c8fb41ecca9dbd85abd3e625d453690cd2e2d25ccf8ceaca4ae5e67f6b3249c0
Sep 09 20:49:06.566081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:06.566051    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:06.566187 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:06.566169    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:06.566256 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:06.566242    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:49:07.566220578 +0000 UTC m=+36.069050577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:07.216708 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:07.216630    2548 generic.go:356] "Generic (PLEG): container finished" podID="7cb2c318-c35f-4b72-a32c-7d0371d26552" containerID="9bfa37916fea035195ad0ddeab9bd2b2a319348beb0a78835d03992f9dc21208" exitCode=0
Sep 09 20:49:07.217288 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:07.216715    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerDied","Data":"9bfa37916fea035195ad0ddeab9bd2b2a319348beb0a78835d03992f9dc21208"}
Sep 09 20:49:07.217830 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:07.217811    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xcvt7" event={"ID":"086ea7e4-5ed3-41cc-9842-287c26d02fc7","Type":"ContainerStarted","Data":"c8fb41ecca9dbd85abd3e625d453690cd2e2d25ccf8ceaca4ae5e67f6b3249c0"}
Sep 09 20:49:07.575067 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:07.574990    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:07.575207 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:07.575168    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:07.575258 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:07.575248    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:49:09.575225669 +0000 UTC m=+38.078055661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:08.222752 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:08.222705    2548 generic.go:356] "Generic (PLEG): container finished" podID="7cb2c318-c35f-4b72-a32c-7d0371d26552" containerID="1a449bee22d3b852c8e2692ebbc505a4c8b801594525832f67971d56464c6f4d" exitCode=0
Sep 09 20:49:08.223169 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:08.222770    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerDied","Data":"1a449bee22d3b852c8e2692ebbc505a4c8b801594525832f67971d56464c6f4d"}
Sep 09 20:49:09.228462 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:09.228424    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g45br" event={"ID":"7cb2c318-c35f-4b72-a32c-7d0371d26552","Type":"ContainerStarted","Data":"96faa61a1bea67bbccbaf0415d002c7a400790d903e343b703dec91a3ddcd12c"}
Sep 09 20:49:09.250859 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:09.250810    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g45br" podStartSLOduration=9.210816695 podStartE2EDuration="37.250796629s" totalImagesPullingTime="28.039979934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:09.250469916 +0000 UTC m=+37.753299919" watchObservedRunningTime="2026-09-09 20:49:09.250796629 +0000 UTC m=+37.753626629"
Sep 09 20:49:09.591214 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:09.590996    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:09.591214 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:09.591147    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:09.591384 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:09.591221    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:49:13.591206481 +0000 UTC m=+42.094036460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:10.232181 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:10.232078    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xcvt7" event={"ID":"086ea7e4-5ed3-41cc-9842-287c26d02fc7","Type":"ContainerStarted","Data":"018db8e2f1a5590828fc9dc38c9fc9cce8f92fdb0f26e69c790754314ec26ab8"}
Sep 09 20:49:13.617515 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:13.617463    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:13.617911 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:13.617584    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:13.617911 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:13.617635    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:49:21.617621983 +0000 UTC m=+50.120451961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:21.670283 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:21.670241    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:21.670693 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:21.670402    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:21.670693 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:21.670464    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:49:37.670445596 +0000 UTC m=+66.173275587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:32.213367 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:32.213328    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rsp5l"
Sep 09 20:49:32.241331 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:32.241281    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xcvt7" podStartSLOduration=24.447400098 podStartE2EDuration="27.241267196s" totalImagesPullingTime="2.793867098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:10.245511859 +0000 UTC m=+38.748341861" watchObservedRunningTime="2026-09-09 20:49:32.241267196 +0000 UTC m=+60.744097229"
Sep 09 20:49:37.671004 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.670954    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:49:37.671527 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.671030    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:49:37.671527 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:37.671121    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:49:37.671527 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:37.671183    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:50:09.671168761 +0000 UTC m=+98.173998740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:49:37.673624 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.673600    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:49:37.681875 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:37.681850    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:49:37.682000 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:49:37.681931    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:41.681910065 +0000 UTC m=+130.184740044 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : secret "metrics-daemon-secret" not found
Sep 09 20:49:37.771794 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.771757    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:37.774065 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.774048    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:49:37.785632 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.785598    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/813abdcc-2cfe-4ba7-95d4-42fce7903b66-original-pull-secret\") pod \"global-pull-secret-syncer-78qgz\" (UID: \"813abdcc-2cfe-4ba7-95d4-42fce7903b66\") " pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:37.787522 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.787496    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-78qgz"
Sep 09 20:49:37.872784 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.872747    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:37.875147 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.875129    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:49:37.885090 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.885072    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:49:37.896169 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:37.896146    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thtgb\" (UniqueName: \"kubernetes.io/projected/23cb1fd8-9dda-4c9b-8e99-513654529705-kube-api-access-thtgb\") pod \"network-check-target-dx5nv\" (UID: \"23cb1fd8-9dda-4c9b-8e99-513654529705\") " pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:38.057676 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:38.057650    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-78qgz"]
Sep 09 20:49:38.060558 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:49:38.060530    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813abdcc_2cfe_4ba7_95d4_42fce7903b66.slice/crio-e2e2b6e93e60b524d8af9d99c7c1aec06c9de68ab7c9a6512a62056720cb976b WatchSource:0}: Error finding container e2e2b6e93e60b524d8af9d99c7c1aec06c9de68ab7c9a6512a62056720cb976b: Status 404 returned error can't find the container with id e2e2b6e93e60b524d8af9d99c7c1aec06c9de68ab7c9a6512a62056720cb976b
Sep 09 20:49:38.084195 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:38.084171    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m7n9z\""
Sep 09 20:49:38.092683 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:38.092667    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:38.300530 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:38.300502    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-78qgz" event={"ID":"813abdcc-2cfe-4ba7-95d4-42fce7903b66","Type":"ContainerStarted","Data":"e2e2b6e93e60b524d8af9d99c7c1aec06c9de68ab7c9a6512a62056720cb976b"}
Sep 09 20:49:38.359426 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:38.359371    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dx5nv"]
Sep 09 20:49:38.362380 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:49:38.362358    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb1fd8_9dda_4c9b_8e99_513654529705.slice/crio-48fd593d62e4c6329c06912d9558fb9e1b65c181a78b3cf98dd99210846eeaca WatchSource:0}: Error finding container 48fd593d62e4c6329c06912d9558fb9e1b65c181a78b3cf98dd99210846eeaca: Status 404 returned error can't find the container with id 48fd593d62e4c6329c06912d9558fb9e1b65c181a78b3cf98dd99210846eeaca
Sep 09 20:49:39.304814 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:39.304771    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dx5nv" event={"ID":"23cb1fd8-9dda-4c9b-8e99-513654529705","Type":"ContainerStarted","Data":"48fd593d62e4c6329c06912d9558fb9e1b65c181a78b3cf98dd99210846eeaca"}
Sep 09 20:49:43.316496 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:43.316454    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-78qgz" event={"ID":"813abdcc-2cfe-4ba7-95d4-42fce7903b66","Type":"ContainerStarted","Data":"2b600ac8180fa14ab450b9449339981ece6a12d3621c74f1d9f4cb363cfdc9c4"}
Sep 09 20:49:43.317930 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:43.317906    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dx5nv" event={"ID":"23cb1fd8-9dda-4c9b-8e99-513654529705","Type":"ContainerStarted","Data":"95fa52cae5738a34a1141ffe992ca522452567d95a429bb35e9af86173035837"}
Sep 09 20:49:43.318054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:43.318023    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:49:43.331128 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:43.331084    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-78qgz" podStartSLOduration=66.675369585 podStartE2EDuration="1m11.331069992s" totalImagesPullingTime="4.655700407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:43.330247096 +0000 UTC m=+71.833077102" watchObservedRunningTime="2026-09-09 20:49:43.331069992 +0000 UTC m=+71.833899987"
Sep 09 20:49:43.347485 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:49:43.347449    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dx5nv" podStartSLOduration=66.998833878 podStartE2EDuration="1m11.347435865s" totalImagesPullingTime="4.348601987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:43.347418808 +0000 UTC m=+71.850248806" watchObservedRunningTime="2026-09-09 20:49:43.347435865 +0000 UTC m=+71.850265849"
Sep 09 20:50:09.684785 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:09.684643    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:50:09.685251 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:50:09.684831    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:50:09.685251 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:50:09.684922    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:51:13.684903951 +0000 UTC m=+162.187733934 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:50:14.322814 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:14.322784    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dx5nv"
Sep 09 20:50:15.175420 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.175381    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"]
Sep 09 20:50:15.210011 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.209974    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"]
Sep 09 20:50:15.210164 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.210098    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.212045 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.212024    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-455q4\""
Sep 09 20:50:15.212928 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.212909    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:50:15.213020 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.212932    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:50:15.213020 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.212908    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:50:15.213020 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.212965    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:50:15.317621 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.317579    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9730d2f-4162-43d1-aa6f-507a893845a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866769ff6d-v9s22\" (UID: \"b9730d2f-4162-43d1-aa6f-507a893845a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.317816 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.317636    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp29j\" (UniqueName: \"kubernetes.io/projected/b9730d2f-4162-43d1-aa6f-507a893845a3-kube-api-access-qp29j\") pod \"managed-serviceaccount-addon-agent-866769ff6d-v9s22\" (UID: \"b9730d2f-4162-43d1-aa6f-507a893845a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.418796 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.418767    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9730d2f-4162-43d1-aa6f-507a893845a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866769ff6d-v9s22\" (UID: \"b9730d2f-4162-43d1-aa6f-507a893845a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.419094 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.418819    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp29j\" (UniqueName: \"kubernetes.io/projected/b9730d2f-4162-43d1-aa6f-507a893845a3-kube-api-access-qp29j\") pod \"managed-serviceaccount-addon-agent-866769ff6d-v9s22\" (UID: \"b9730d2f-4162-43d1-aa6f-507a893845a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.421211 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.421190    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9730d2f-4162-43d1-aa6f-507a893845a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866769ff6d-v9s22\" (UID: \"b9730d2f-4162-43d1-aa6f-507a893845a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.426952 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.426899    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp29j\" (UniqueName: \"kubernetes.io/projected/b9730d2f-4162-43d1-aa6f-507a893845a3-kube-api-access-qp29j\") pod \"managed-serviceaccount-addon-agent-866769ff6d-v9s22\" (UID: \"b9730d2f-4162-43d1-aa6f-507a893845a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.531162 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.531122    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"
Sep 09 20:50:15.814273 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:15.814235    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22"]
Sep 09 20:50:15.818323 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:50:15.818296    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9730d2f_4162_43d1_aa6f_507a893845a3.slice/crio-ebafbb61ffc628c03dbb89270bb94c9dbd43fbb2152879b329a8548a90553263 WatchSource:0}: Error finding container ebafbb61ffc628c03dbb89270bb94c9dbd43fbb2152879b329a8548a90553263: Status 404 returned error can't find the container with id ebafbb61ffc628c03dbb89270bb94c9dbd43fbb2152879b329a8548a90553263
Sep 09 20:50:16.397336 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:16.397299    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22" event={"ID":"b9730d2f-4162-43d1-aa6f-507a893845a3","Type":"ContainerStarted","Data":"ebafbb61ffc628c03dbb89270bb94c9dbd43fbb2152879b329a8548a90553263"}
Sep 09 20:50:19.406589 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:19.406555    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22" event={"ID":"b9730d2f-4162-43d1-aa6f-507a893845a3","Type":"ContainerStarted","Data":"8f09dd7e5bf344800de28d7f9a3cfed75f93283d89615e19bd3c8f1fe2d46146"}
Sep 09 20:50:19.420675 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:19.420613    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22" podStartSLOduration=1.7873372010000002 podStartE2EDuration="4.420593754s" totalImagesPullingTime="2.633256553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:19.420349478 +0000 UTC m=+107.923179480" watchObservedRunningTime="2026-09-09 20:50:19.420593754 +0000 UTC m=+107.923423756"
Sep 09 20:50:41.703962 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:50:41.703909    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:50:41.704445 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:50:41.704069    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:50:41.704445 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:50:41.704152    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:43.704133355 +0000 UTC m=+252.206963335 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : secret "metrics-daemon-secret" not found
Sep 09 20:51:08.861321 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:51:08.861259    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wl49r" podUID="ce390400-86f1-4cf0-8155-8313e802c2da"
Sep 09 20:51:08.975992 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:51:08.975953    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:51:09.520978 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:51:09.520945    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wl49r"
Sep 09 20:51:13.725811 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:51:13.725775    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:51:13.726273 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:51:13.725879    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:51:13.726273 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:51:13.725944    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls podName:ce390400-86f1-4cf0-8155-8313e802c2da nodeName:}" failed. No retries permitted until 2026-09-09 20:53:15.725926838 +0000 UTC m=+284.228756831 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls") pod "dns-default-wl49r" (UID: "ce390400-86f1-4cf0-8155-8313e802c2da") : secret "dns-default-metrics-tls" not found
Sep 09 20:51:22.962375 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:51:22.962337    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:52:18.688229 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:18.688193    2548 generic.go:356] "Generic (PLEG): container finished" podID="b9730d2f-4162-43d1-aa6f-507a893845a3" containerID="8f09dd7e5bf344800de28d7f9a3cfed75f93283d89615e19bd3c8f1fe2d46146" exitCode=255
Sep 09 20:52:18.688618 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:18.688245    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22" event={"ID":"b9730d2f-4162-43d1-aa6f-507a893845a3","Type":"ContainerDied","Data":"8f09dd7e5bf344800de28d7f9a3cfed75f93283d89615e19bd3c8f1fe2d46146"}
Sep 09 20:52:18.693556 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:18.693535    2548 scope.go:117] "RemoveContainer" containerID="8f09dd7e5bf344800de28d7f9a3cfed75f93283d89615e19bd3c8f1fe2d46146"
Sep 09 20:52:19.691817 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:19.691782    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866769ff6d-v9s22" event={"ID":"b9730d2f-4162-43d1-aa6f-507a893845a3","Type":"ContainerStarted","Data":"0635ccdc1723e205892773d92f112c7a87f5b84e2c5a862bb1d7ca586f8fb3fb"}
Sep 09 20:52:43.727725 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:43.727669    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:52:43.728227 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:43.727832    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:52:43.728227 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:43.727901    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs podName:870cf306-82d5-4437-9477-5c5d924506e8 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:45.727884362 +0000 UTC m=+374.230714341 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs") pod "network-metrics-daemon-5hfr5" (UID: "870cf306-82d5-4437-9477-5c5d924506e8") : secret "metrics-daemon-secret" not found
Sep 09 20:52:55.046650 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.046557    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6564b7c66d-qpftd"]
Sep 09 20:52:55.049495 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.049474    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.052819 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.052789    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:52:55.052819 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.052813    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:52:55.052999 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.052828    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:52:55.053180 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.053159    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:52:55.053314 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.053298    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:52:55.053314 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.053309    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:52:55.053480 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.053411    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-q5jj4\""
Sep 09 20:52:55.063724 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.063698    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6564b7c66d-qpftd"]
Sep 09 20:52:55.140837 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.140788    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"]
Sep 09 20:52:55.143540 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.143518    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fb6cz"]
Sep 09 20:52:55.143674 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.143634    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.146179 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.146162    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"]
Sep 09 20:52:55.146294 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.146276    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.146367 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.146348    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:52:55.146455 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.146354    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:52:55.147245 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.147231    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:52:55.147323 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.147243    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rqhp4\""
Sep 09 20:52:55.147323 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.147259    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:52:55.147865 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.147848    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:52:55.148885 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.148870    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.150679 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.150661    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:55.150935 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.150920    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d6l82\""
Sep 09 20:52:55.151159 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.151145    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:55.151230 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.151178    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:52:55.152559 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.152542    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:52:55.152833 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.152815    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:52:55.153064 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.153047    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-smmqf\""
Sep 09 20:52:55.153241 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.153223    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:52:55.159748 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.159705    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fb6cz"]
Sep 09 20:52:55.161484 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.161459    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:52:55.165260 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.165241    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:52:55.166438 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.166419    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"]
Sep 09 20:52:55.167382 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.167364    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"]
Sep 09 20:52:55.203875 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.203848    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.203981 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.203895    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzssq\" (UniqueName: \"kubernetes.io/projected/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-kube-api-access-nzssq\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.203981 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.203916    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.203981 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.203939    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.203981 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.203961    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-stats-auth\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.204111 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.203993    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb8m\" (UniqueName: \"kubernetes.io/projected/1d6d21b7-47d9-423f-9127-f52cec9a1903-kube-api-access-ddb8m\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.204111 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.204011    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-default-certificate\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.204111 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.204029    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1d6d21b7-47d9-423f-9127-f52cec9a1903-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.305046 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.304958    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.305046 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.304999    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be85228e-37c3-4a62-8108-7068b40b7881-ca-trust-extracted\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305046 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305016    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-bound-sa-token\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305051    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzssq\" (UniqueName: \"kubernetes.io/projected/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-kube-api-access-nzssq\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305076    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305103    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-image-registry-private-configuration\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.305105    2548 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305124    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-trusted-ca\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305143    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305167    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-stats-auth\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.305180    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.805162569 +0000 UTC m=+264.307992553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : secret "router-metrics-certs-default" not found
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.305199    2548 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.305265    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls podName:1d6d21b7-47d9-423f-9127-f52cec9a1903 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.805250048 +0000 UTC m=+264.308080042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-49fbc" (UID: "1d6d21b7-47d9-423f-9127-f52cec9a1903") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.305283    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.805275033 +0000 UTC m=+264.308105012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:55.305322 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305261    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-config\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305329    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5xz\" (UniqueName: \"kubernetes.io/projected/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-kube-api-access-8d5xz\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305389    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5pm2\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-kube-api-access-l5pm2\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305450    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-registry-certificates\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305494    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb8m\" (UniqueName: \"kubernetes.io/projected/1d6d21b7-47d9-423f-9127-f52cec9a1903-kube-api-access-ddb8m\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305515    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-trusted-ca\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305531    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305548    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-default-certificate\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305564    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1d6d21b7-47d9-423f-9127-f52cec9a1903-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305587    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-serving-cert\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.305856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.305607    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-installation-pull-secrets\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.306199 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.306148    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1d6d21b7-47d9-423f-9127-f52cec9a1903-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.307645 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.307624    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-stats-auth\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.307879 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.307861    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-default-certificate\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.317456 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.317424    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb8m\" (UniqueName: \"kubernetes.io/projected/1d6d21b7-47d9-423f-9127-f52cec9a1903-kube-api-access-ddb8m\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.317553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.317481    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzssq\" (UniqueName: \"kubernetes.io/projected/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-kube-api-access-nzssq\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.406846 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.406807    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-registry-certificates\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.406846 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.406845    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-trusted-ca\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.406861    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.406946    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.406958    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8: secret "image-registry-tls" not found
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.406995    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-serving-cert\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.407023    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls podName:be85228e-37c3-4a62-8108-7068b40b7881 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:55.907004434 +0000 UTC m=+264.409834414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls") pod "image-registry-6bb6b9dcbd-bfqm8" (UID: "be85228e-37c3-4a62-8108-7068b40b7881") : secret "image-registry-tls" not found
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407050    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-installation-pull-secrets\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407135 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407123    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be85228e-37c3-4a62-8108-7068b40b7881-ca-trust-extracted\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407150    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-bound-sa-token\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407208    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-image-registry-private-configuration\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407245    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-trusted-ca\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407300    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-config\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407324    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5xz\" (UniqueName: \"kubernetes.io/projected/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-kube-api-access-8d5xz\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407358    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5pm2\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-kube-api-access-l5pm2\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407442    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-registry-certificates\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407470    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be85228e-37c3-4a62-8108-7068b40b7881-ca-trust-extracted\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.407896 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.407809    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-trusted-ca\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.408114 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.408088    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-config\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.408439 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.408414    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-trusted-ca\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.409630 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.409604    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-installation-pull-secrets\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.409726 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.409682    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-image-registry-private-configuration\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.409803 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.409788    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-serving-cert\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.417493 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.417468    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5pm2\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-kube-api-access-l5pm2\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.417622 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.417605    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-bound-sa-token\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.417881 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.417862    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5xz\" (UniqueName: \"kubernetes.io/projected/1c71f27f-50a9-42a4-b6dd-21c58a14e5a3-kube-api-access-8d5xz\") pod \"console-operator-85b454874b-fb6cz\" (UID: \"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3\") " pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.465095 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.465047    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:52:55.734889 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.734856    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fb6cz"]
Sep 09 20:52:55.738438 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:52:55.738409    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c71f27f_50a9_42a4_b6dd_21c58a14e5a3.slice/crio-7c6fca49dbf4f6b97752371cf0e0def3b45b1f6593a28abef0dd1a20616a0053 WatchSource:0}: Error finding container 7c6fca49dbf4f6b97752371cf0e0def3b45b1f6593a28abef0dd1a20616a0053: Status 404 returned error can't find the container with id 7c6fca49dbf4f6b97752371cf0e0def3b45b1f6593a28abef0dd1a20616a0053
Sep 09 20:52:55.776690 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.776654    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" event={"ID":"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3","Type":"ContainerStarted","Data":"7c6fca49dbf4f6b97752371cf0e0def3b45b1f6593a28abef0dd1a20616a0053"}
Sep 09 20:52:55.810614 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.810582    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:55.810797 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.810627    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.810797 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.810722    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:55.810797 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.810761    2548 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:55.810922 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.810823    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls podName:1d6d21b7-47d9-423f-9127-f52cec9a1903 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.810804956 +0000 UTC m=+265.313634958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-49fbc" (UID: "1d6d21b7-47d9-423f-9127-f52cec9a1903") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:55.810922 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.810846    2548 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:55.810922 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.810856    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.810840746 +0000 UTC m=+265.313670725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:55.810922 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.810880    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.81087399 +0000 UTC m=+265.313703969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : secret "router-metrics-certs-default" not found
Sep 09 20:52:55.911237 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:55.911206    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:55.911367 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.911322    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:55.911367 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.911335    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8: secret "image-registry-tls" not found
Sep 09 20:52:55.911458 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:55.911385    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls podName:be85228e-37c3-4a62-8108-7068b40b7881 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.911369624 +0000 UTC m=+265.414199603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls") pod "image-registry-6bb6b9dcbd-bfqm8" (UID: "be85228e-37c3-4a62-8108-7068b40b7881") : secret "image-registry-tls" not found
Sep 09 20:52:56.818990 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:56.818946    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:56.819437 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.819127    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.81910194 +0000 UTC m=+267.321931924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:56.819437 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:56.819233    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:56.819437 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:56.819280    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:56.819437 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.819387    2548 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:56.819437 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.819436    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls podName:1d6d21b7-47d9-423f-9127-f52cec9a1903 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.819424966 +0000 UTC m=+267.322254950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-49fbc" (UID: "1d6d21b7-47d9-423f-9127-f52cec9a1903") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:56.819653 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.819387    2548 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:56.819653 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.819480    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.819471182 +0000 UTC m=+267.322301166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : secret "router-metrics-certs-default" not found
Sep 09 20:52:56.920044 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:56.919983    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:56.920233 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.920115    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:56.920233 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.920127    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8: secret "image-registry-tls" not found
Sep 09 20:52:56.920233 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:56.920179    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls podName:be85228e-37c3-4a62-8108-7068b40b7881 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:58.920166583 +0000 UTC m=+267.422996561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls") pod "image-registry-6bb6b9dcbd-bfqm8" (UID: "be85228e-37c3-4a62-8108-7068b40b7881") : secret "image-registry-tls" not found
Sep 09 20:52:58.786546 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.786514    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/0.log"
Sep 09 20:52:58.787030 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.786557    2548 generic.go:356] "Generic (PLEG): container finished" podID="1c71f27f-50a9-42a4-b6dd-21c58a14e5a3" containerID="8c911ae716de97293854dbc5d31a65dd1ead564e468bd0b34bb2d9167c9925a2" exitCode=255
Sep 09 20:52:58.787030 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.786595    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" event={"ID":"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3","Type":"ContainerDied","Data":"8c911ae716de97293854dbc5d31a65dd1ead564e468bd0b34bb2d9167c9925a2"}
Sep 09 20:52:58.787030 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.786967    2548 scope.go:117] "RemoveContainer" containerID="8c911ae716de97293854dbc5d31a65dd1ead564e468bd0b34bb2d9167c9925a2"
Sep 09 20:52:58.834685 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.834648    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:58.834853 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.834707    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:52:58.834853 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.834788    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:52:58.834853 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.834828    2548 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:58.834997 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.834892    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls podName:1d6d21b7-47d9-423f-9127-f52cec9a1903 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.834876553 +0000 UTC m=+271.337706532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-49fbc" (UID: "1d6d21b7-47d9-423f-9127-f52cec9a1903") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:58.834997 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.834828    2548 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:52:58.834997 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.834906    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.834899671 +0000 UTC m=+271.337729650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : configmap references non-existent config key: service-ca.crt
Sep 09 20:52:58.834997 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.834933    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.834916232 +0000 UTC m=+271.337746213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : secret "router-metrics-certs-default" not found
Sep 09 20:52:58.935419 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:58.935380    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:52:58.935613 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.935526    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:58.935613 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.935553    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8: secret "image-registry-tls" not found
Sep 09 20:52:58.935613 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:58.935614    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls podName:be85228e-37c3-4a62-8108-7068b40b7881 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.935593749 +0000 UTC m=+271.438423762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls") pod "image-registry-6bb6b9dcbd-bfqm8" (UID: "be85228e-37c3-4a62-8108-7068b40b7881") : secret "image-registry-tls" not found
Sep 09 20:52:59.792125 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:59.792098    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:52:59.792517 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:59.792494    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/0.log"
Sep 09 20:52:59.792559 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:59.792527    2548 generic.go:356] "Generic (PLEG): container finished" podID="1c71f27f-50a9-42a4-b6dd-21c58a14e5a3" containerID="496628307e1612b2d5b4bc30d17bd0a229e756e4649e69b709abb710e4d07e0f" exitCode=255
Sep 09 20:52:59.792595 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:59.792557    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" event={"ID":"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3","Type":"ContainerDied","Data":"496628307e1612b2d5b4bc30d17bd0a229e756e4649e69b709abb710e4d07e0f"}
Sep 09 20:52:59.792636 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:59.792623    2548 scope.go:117] "RemoveContainer" containerID="8c911ae716de97293854dbc5d31a65dd1ead564e468bd0b34bb2d9167c9925a2"
Sep 09 20:52:59.792883 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:52:59.792864    2548 scope.go:117] "RemoveContainer" containerID="496628307e1612b2d5b4bc30d17bd0a229e756e4649e69b709abb710e4d07e0f"
Sep 09 20:52:59.793042 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:52:59.793021    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fb6cz_openshift-console-operator(1c71f27f-50a9-42a4-b6dd-21c58a14e5a3)\"" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" podUID="1c71f27f-50a9-42a4-b6dd-21c58a14e5a3"
Sep 09 20:53:00.244562 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.244532    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"]
Sep 09 20:53:00.248411 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.248395    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"
Sep 09 20:53:00.252724 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.252701    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4plfj\""
Sep 09 20:53:00.252839 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.252700    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 20:53:00.253407 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.253389    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 20:53:00.262288 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.262268    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"]
Sep 09 20:53:00.345265 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.345212    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxz7\" (UniqueName: \"kubernetes.io/projected/67f23aa7-d4b5-4d14-89a8-b1116f8c47b0-kube-api-access-6wxz7\") pod \"migrator-fdbb8fdff-tvf27\" (UID: \"67f23aa7-d4b5-4d14-89a8-b1116f8c47b0\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"
Sep 09 20:53:00.395512 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.395480    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vkrpg"]
Sep 09 20:53:00.398484 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.398469    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.401249 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.401226    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:53:00.401369 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.401317    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:53:00.401369 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.401314    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:53:00.401369 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.401361    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngfdf\""
Sep 09 20:53:00.402188 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.402172    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:53:00.421331 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.421301    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkrpg"]
Sep 09 20:53:00.446618 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.446588    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxz7\" (UniqueName: \"kubernetes.io/projected/67f23aa7-d4b5-4d14-89a8-b1116f8c47b0-kube-api-access-6wxz7\") pod \"migrator-fdbb8fdff-tvf27\" (UID: \"67f23aa7-d4b5-4d14-89a8-b1116f8c47b0\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"
Sep 09 20:53:00.456120 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.456093    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxz7\" (UniqueName: \"kubernetes.io/projected/67f23aa7-d4b5-4d14-89a8-b1116f8c47b0-kube-api-access-6wxz7\") pod \"migrator-fdbb8fdff-tvf27\" (UID: \"67f23aa7-d4b5-4d14-89a8-b1116f8c47b0\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"
Sep 09 20:53:00.547255 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.547167    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzksj\" (UniqueName: \"kubernetes.io/projected/73d72e45-d692-4885-8d99-6fdb2f027312-kube-api-access-nzksj\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.547255 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.547222    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73d72e45-d692-4885-8d99-6fdb2f027312-data-volume\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.547255 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.547244    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.547441 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.547320    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73d72e45-d692-4885-8d99-6fdb2f027312-crio-socket\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.547441 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.547351    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.556270 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.556252    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.648842    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzksj\" (UniqueName: \"kubernetes.io/projected/73d72e45-d692-4885-8d99-6fdb2f027312-kube-api-access-nzksj\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.648930    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73d72e45-d692-4885-8d99-6fdb2f027312-data-volume\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.648964    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.649017    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73d72e45-d692-4885-8d99-6fdb2f027312-crio-socket\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.649056    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:00.649313    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:00.649388    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.149369806 +0000 UTC m=+269.652199800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.650240    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/73d72e45-d692-4885-8d99-6fdb2f027312-data-volume\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:00.650375    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:00.650429    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:01.150413813 +0000 UTC m=+269.653243808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:00.652193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.650495    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/73d72e45-d692-4885-8d99-6fdb2f027312-crio-socket\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.660677 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.660655    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzksj\" (UniqueName: \"kubernetes.io/projected/73d72e45-d692-4885-8d99-6fdb2f027312-kube-api-access-nzksj\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:00.798636 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.798554    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:53:00.799019 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.798908    2548 scope.go:117] "RemoveContainer" containerID="496628307e1612b2d5b4bc30d17bd0a229e756e4649e69b709abb710e4d07e0f"
Sep 09 20:53:00.799067 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:00.799049    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fb6cz_openshift-console-operator(1c71f27f-50a9-42a4-b6dd-21c58a14e5a3)\"" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" podUID="1c71f27f-50a9-42a4-b6dd-21c58a14e5a3"
Sep 09 20:53:00.820696 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:00.820659    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27"]
Sep 09 20:53:00.824251 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:00.824228    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f23aa7_d4b5_4d14_89a8_b1116f8c47b0.slice/crio-73a7dba43db24e1bd2ab6524060ea382ea1a3f4dd47dc5a66b77c1dc0273c144 WatchSource:0}: Error finding container 73a7dba43db24e1bd2ab6524060ea382ea1a3f4dd47dc5a66b77c1dc0273c144: Status 404 returned error can't find the container with id 73a7dba43db24e1bd2ab6524060ea382ea1a3f4dd47dc5a66b77c1dc0273c144
Sep 09 20:53:01.153796 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:01.153703    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:01.153796 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:01.153773    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:01.153983 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:01.153814    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:01.153983 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:01.153879    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.153863227 +0000 UTC m=+270.656693207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:01.153983 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:01.153879    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:01.153983 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:01.153920    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:02.153908558 +0000 UTC m=+270.656738538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:01.803029 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:01.802994    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27" event={"ID":"67f23aa7-d4b5-4d14-89a8-b1116f8c47b0","Type":"ContainerStarted","Data":"73a7dba43db24e1bd2ab6524060ea382ea1a3f4dd47dc5a66b77c1dc0273c144"}
Sep 09 20:53:02.160606 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.160503    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:02.160786 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.160616    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:02.160786 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.160640    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:02.160786 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.160696    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.160673577 +0000 UTC m=+272.663503559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:02.160786 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.160774    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:02.161022 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.160825    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:04.160811835 +0000 UTC m=+272.663641820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:02.807857 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.807823    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27" event={"ID":"67f23aa7-d4b5-4d14-89a8-b1116f8c47b0","Type":"ContainerStarted","Data":"f7884cf35036f2d86d8cd7445fc1d7550adae694245cee7b29403c403e7ce952"}
Sep 09 20:53:02.808225 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.807874    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27" event={"ID":"67f23aa7-d4b5-4d14-89a8-b1116f8c47b0","Type":"ContainerStarted","Data":"f833d8330815049e1b1837c3a04ce2a5fb25518091ff2d35588b41003e5271d5"}
Sep 09 20:53:02.829795 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.829745    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-tvf27" podStartSLOduration=1.320852307 podStartE2EDuration="2.829713932s" totalImagesPullingTime="1.508861625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.828993355 +0000 UTC m=+271.331823354" watchObservedRunningTime="2026-09-09 20:53:02.829713932 +0000 UTC m=+271.332543942"
Sep 09 20:53:02.866812 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.866774    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:02.866986 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.866836    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:02.866986 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.866892    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:53:02.866986 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.866937    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.866914124 +0000 UTC m=+279.369744108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : configmap references non-existent config key: service-ca.crt
Sep 09 20:53:02.867134 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.866992    2548 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:02.867134 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.867000    2548 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:53:02.867134 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.867031    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls podName:1d6d21b7-47d9-423f-9127-f52cec9a1903 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.867020948 +0000 UTC m=+279.369850927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-49fbc" (UID: "1d6d21b7-47d9-423f-9127-f52cec9a1903") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:02.867134 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.867052    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs podName:8100cb47-78a6-4eb4-9bfb-ccdc3025876f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.867036786 +0000 UTC m=+279.369866779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs") pod "router-default-6564b7c66d-qpftd" (UID: "8100cb47-78a6-4eb4-9bfb-ccdc3025876f") : secret "router-metrics-certs-default" not found
Sep 09 20:53:02.962154 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.962131    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dkcxd_9dc7b235-2a09-46a5-9165-39e41d98d840/dns-node-resolver/0.log"
Sep 09 20:53:02.968116 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:02.968087    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:02.968256 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.968235    2548 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:53:02.968256 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.968250    2548 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8: secret "image-registry-tls" not found
Sep 09 20:53:02.968357 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:02.968316    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls podName:be85228e-37c3-4a62-8108-7068b40b7881 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:10.96829598 +0000 UTC m=+279.471125976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls") pod "image-registry-6bb6b9dcbd-bfqm8" (UID: "be85228e-37c3-4a62-8108-7068b40b7881") : secret "image-registry-tls" not found
Sep 09 20:53:04.152583 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:04.152550    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nsh2j_6619b496-637a-4621-a857-75c2fc6c97be/node-ca/0.log"
Sep 09 20:53:04.177080 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:04.177041    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:04.177244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:04.177156    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:04.177244 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:04.177217    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:04.177244 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:04.177229    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.177389 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:04.177281    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:08.17726722 +0000 UTC m=+276.680097200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:04.177389 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:04.177295    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:08.177288618 +0000 UTC m=+276.680118596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.141962 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:05.141926    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xcvt7_086ea7e4-5ed3-41cc-9842-287c26d02fc7/serve-healthcheck-canary/0.log"
Sep 09 20:53:05.342447 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:05.342415    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-tvf27_67f23aa7-d4b5-4d14-89a8-b1116f8c47b0/migrator/0.log"
Sep 09 20:53:05.465321 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:05.465288    2548 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:53:05.465476 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:05.465352    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:53:05.465705 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:05.465693    2548 scope.go:117] "RemoveContainer" containerID="496628307e1612b2d5b4bc30d17bd0a229e756e4649e69b709abb710e4d07e0f"
Sep 09 20:53:05.465894 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:05.465875    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fb6cz_openshift-console-operator(1c71f27f-50a9-42a4-b6dd-21c58a14e5a3)\"" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" podUID="1c71f27f-50a9-42a4-b6dd-21c58a14e5a3"
Sep 09 20:53:05.542953 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:05.542923    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-tvf27_67f23aa7-d4b5-4d14-89a8-b1116f8c47b0/graceful-termination/0.log"
Sep 09 20:53:08.212959 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:08.212924    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:08.213344 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:08.212976    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:08.213344 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:08.213037    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:53:08.213344 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:08.213095    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.213080119 +0000 UTC m=+284.715910097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : configmap "kube-rbac-proxy" not found
Sep 09 20:53:08.213344 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:08.213119    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:08.213344 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:08.213158    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls podName:73d72e45-d692-4885-8d99-6fdb2f027312 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.213146971 +0000 UTC m=+284.715976950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkrpg" (UID: "73d72e45-d692-4885-8d99-6fdb2f027312") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:10.936775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:10.936717    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:53:10.937192 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:10.936853    2548 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:10.937192 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:10.936854    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:10.937192 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:10.936933    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls podName:1d6d21b7-47d9-423f-9127-f52cec9a1903 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:26.936918987 +0000 UTC m=+295.439748969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-49fbc" (UID: "1d6d21b7-47d9-423f-9127-f52cec9a1903") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:10.937192 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:10.936989    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:10.937374 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:10.937357    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-service-ca-bundle\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:10.939211 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:10.939194    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8100cb47-78a6-4eb4-9bfb-ccdc3025876f-metrics-certs\") pod \"router-default-6564b7c66d-qpftd\" (UID: \"8100cb47-78a6-4eb4-9bfb-ccdc3025876f\") " pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:10.958040 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:10.958017    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:11.037348 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.037318    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:11.039551 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.039526    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"image-registry-6bb6b9dcbd-bfqm8\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") " pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:11.070279 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.070256    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:11.229170 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.229126    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6564b7c66d-qpftd"]
Sep 09 20:53:11.231498 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:11.231468    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8100cb47_78a6_4eb4_9bfb_ccdc3025876f.slice/crio-879fa5c69c8e3c60211738acb838ddf8507ea844e63068562c35aca1805e96d9 WatchSource:0}: Error finding container 879fa5c69c8e3c60211738acb838ddf8507ea844e63068562c35aca1805e96d9: Status 404 returned error can't find the container with id 879fa5c69c8e3c60211738acb838ddf8507ea844e63068562c35aca1805e96d9
Sep 09 20:53:11.339976 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.339944    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"]
Sep 09 20:53:11.343025 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:11.343001    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe85228e_37c3_4a62_8108_7068b40b7881.slice/crio-ab43deba8ce5e05108e9760faf97ed8a64ec560bc19a72d4e6b0e7837ddf5077 WatchSource:0}: Error finding container ab43deba8ce5e05108e9760faf97ed8a64ec560bc19a72d4e6b0e7837ddf5077: Status 404 returned error can't find the container with id ab43deba8ce5e05108e9760faf97ed8a64ec560bc19a72d4e6b0e7837ddf5077
Sep 09 20:53:11.837296 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.837262    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8" event={"ID":"be85228e-37c3-4a62-8108-7068b40b7881","Type":"ContainerStarted","Data":"8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3"}
Sep 09 20:53:11.837462 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.837311    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8" event={"ID":"be85228e-37c3-4a62-8108-7068b40b7881","Type":"ContainerStarted","Data":"ab43deba8ce5e05108e9760faf97ed8a64ec560bc19a72d4e6b0e7837ddf5077"}
Sep 09 20:53:11.837462 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.837381    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:11.838975 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.838952    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6564b7c66d-qpftd" event={"ID":"8100cb47-78a6-4eb4-9bfb-ccdc3025876f","Type":"ContainerStarted","Data":"a280f1f35e8b2db1d34f71a5d3303279b2f22f1b78995b1a8ae4eb140ddbf629"}
Sep 09 20:53:11.839081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.838989    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6564b7c66d-qpftd" event={"ID":"8100cb47-78a6-4eb4-9bfb-ccdc3025876f","Type":"ContainerStarted","Data":"879fa5c69c8e3c60211738acb838ddf8507ea844e63068562c35aca1805e96d9"}
Sep 09 20:53:11.858562 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.858523    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8" podStartSLOduration=16.858510168 podStartE2EDuration="16.858510168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:11.856842292 +0000 UTC m=+280.359672293" watchObservedRunningTime="2026-09-09 20:53:11.858510168 +0000 UTC m=+280.361340168"
Sep 09 20:53:11.875533 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.875494    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6564b7c66d-qpftd" podStartSLOduration=16.875483261 podStartE2EDuration="16.875483261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:11.874516257 +0000 UTC m=+280.377346258" watchObservedRunningTime="2026-09-09 20:53:11.875483261 +0000 UTC m=+280.378313261"
Sep 09 20:53:11.958668 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.958647    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:11.961077 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:11.961060    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:12.521726 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:12.521693    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wl49r" podUID="ce390400-86f1-4cf0-8155-8313e802c2da"
Sep 09 20:53:12.842346 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:12.842256    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wl49r"
Sep 09 20:53:12.842497 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:12.842451    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:12.843299 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:12.843281    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6564b7c66d-qpftd"
Sep 09 20:53:15.772778 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:15.772728    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:53:15.775027 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:15.775004    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce390400-86f1-4cf0-8155-8313e802c2da-metrics-tls\") pod \"dns-default-wl49r\" (UID: \"ce390400-86f1-4cf0-8155-8313e802c2da\") " pod="openshift-dns/dns-default-wl49r"
Sep 09 20:53:15.846185 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:15.846160    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5zq52\""
Sep 09 20:53:15.853685 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:15.853667    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wl49r"
Sep 09 20:53:16.123547 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.123479    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wl49r"]
Sep 09 20:53:16.126462 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:16.126430    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce390400_86f1_4cf0_8155_8313e802c2da.slice/crio-719791eb111e6dc4652b29a0e20316ce7bbb92357e5b924744f47a9ea36f07e9 WatchSource:0}: Error finding container 719791eb111e6dc4652b29a0e20316ce7bbb92357e5b924744f47a9ea36f07e9: Status 404 returned error can't find the container with id 719791eb111e6dc4652b29a0e20316ce7bbb92357e5b924744f47a9ea36f07e9
Sep 09 20:53:16.276250 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.276218    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:16.276408 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.276265    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:16.276718 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.276697    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/73d72e45-d692-4885-8d99-6fdb2f027312-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:16.278824 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.278802    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/73d72e45-d692-4885-8d99-6fdb2f027312-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkrpg\" (UID: \"73d72e45-d692-4885-8d99-6fdb2f027312\") " pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:16.307798 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.307777    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkrpg"
Sep 09 20:53:16.578975 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.578944    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkrpg"]
Sep 09 20:53:16.583426 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:16.582625    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d72e45_d692_4885_8d99_6fdb2f027312.slice/crio-fd1a2fdde2a3a5b3146e74a7bdae814102f02211bd27e3678f6ba0815f1d5905 WatchSource:0}: Error finding container fd1a2fdde2a3a5b3146e74a7bdae814102f02211bd27e3678f6ba0815f1d5905: Status 404 returned error can't find the container with id fd1a2fdde2a3a5b3146e74a7bdae814102f02211bd27e3678f6ba0815f1d5905
Sep 09 20:53:16.855797 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.855705    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkrpg" event={"ID":"73d72e45-d692-4885-8d99-6fdb2f027312","Type":"ContainerStarted","Data":"aeecb7d392526847d5dece7c14323cdaf6a1e6f55b9b98f7905cb9611c2f3915"}
Sep 09 20:53:16.856181 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.855806    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkrpg" event={"ID":"73d72e45-d692-4885-8d99-6fdb2f027312","Type":"ContainerStarted","Data":"fd1a2fdde2a3a5b3146e74a7bdae814102f02211bd27e3678f6ba0815f1d5905"}
Sep 09 20:53:16.856932 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:16.856908    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wl49r" event={"ID":"ce390400-86f1-4cf0-8155-8313e802c2da","Type":"ContainerStarted","Data":"719791eb111e6dc4652b29a0e20316ce7bbb92357e5b924744f47a9ea36f07e9"}
Sep 09 20:53:17.862287 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:17.862258    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkrpg" event={"ID":"73d72e45-d692-4885-8d99-6fdb2f027312","Type":"ContainerStarted","Data":"8dff9d4a0bf7c18417e45dbc02c6329bd3510c2f2e573c8db6136bc322d456db"}
Sep 09 20:53:17.864136 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:17.864114    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wl49r" event={"ID":"ce390400-86f1-4cf0-8155-8313e802c2da","Type":"ContainerStarted","Data":"22b1c12a381857dcc25df355b69a43736a8982368a2ed38b219b0857cacaec9e"}
Sep 09 20:53:17.864234 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:17.864152    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wl49r" event={"ID":"ce390400-86f1-4cf0-8155-8313e802c2da","Type":"ContainerStarted","Data":"b525c41e16aad12993514573292b1168df4222b7ff2a66f0d09c9ff323129180"}
Sep 09 20:53:17.864303 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:17.864248    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wl49r"
Sep 09 20:53:17.881079 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:17.881041    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wl49r" podStartSLOduration=251.312274776 podStartE2EDuration="4m12.881026519s" totalImagesPullingTime="1.568751743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:17.879996253 +0000 UTC m=+286.382826254" watchObservedRunningTime="2026-09-09 20:53:17.881026519 +0000 UTC m=+286.383856510"
Sep 09 20:53:18.962511 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:18.962482    2548 scope.go:117] "RemoveContainer" containerID="496628307e1612b2d5b4bc30d17bd0a229e756e4649e69b709abb710e4d07e0f"
Sep 09 20:53:19.872385 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.872343    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkrpg" event={"ID":"73d72e45-d692-4885-8d99-6fdb2f027312","Type":"ContainerStarted","Data":"51af1909373140870aa62b231cc1fdf55896eb9153e92f03298b06ed2017835a"}
Sep 09 20:53:19.874182 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.874161    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:53:19.874317 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.874210    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" event={"ID":"1c71f27f-50a9-42a4-b6dd-21c58a14e5a3","Type":"ContainerStarted","Data":"b8512bfa1846f0541559c4c3b15d8d0a8bdc73f5e55b5f61c2b0b5b853e75008"}
Sep 09 20:53:19.874463 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.874446    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:53:19.879319 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.879303    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-fb6cz"
Sep 09 20:53:19.892977 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.892938    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vkrpg" podStartSLOduration=17.552327668 podStartE2EDuration="19.892926266s" totalImagesPullingTime="2.340598598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.892203809 +0000 UTC m=+288.395033820" watchObservedRunningTime="2026-09-09 20:53:19.892926266 +0000 UTC m=+288.395756267"
Sep 09 20:53:19.909384 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:19.909342    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-fb6cz" podStartSLOduration=22.876331626 podStartE2EDuration="24.909329201s" totalImagesPullingTime="2.032997575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.908879501 +0000 UTC m=+288.411709502" watchObservedRunningTime="2026-09-09 20:53:19.909329201 +0000 UTC m=+288.412159255"
Sep 09 20:53:25.218185 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.218150    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"]
Sep 09 20:53:25.302396 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.302367    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-749b8b8858-dc7d7"]
Sep 09 20:53:25.305147 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.305132    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.319044 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.319012    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-749b8b8858-dc7d7"]
Sep 09 20:53:25.340499 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340475    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-registry-tls\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340592 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340504    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5hh\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-kube-api-access-8s5hh\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340592 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340541    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-image-registry-private-configuration\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340592 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340566    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-bound-sa-token\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340689 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340602    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-registry-certificates\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340689 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340637    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-trusted-ca\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340689 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340666    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-installation-pull-secrets\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.340813 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.340702    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-ca-trust-extracted\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441121 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441098    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5hh\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-kube-api-access-8s5hh\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441212 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441137    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-image-registry-private-configuration\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441212 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441155    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-bound-sa-token\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441212 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441202    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-registry-certificates\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441364 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441228    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-trusted-ca\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441364 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441275    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-installation-pull-secrets\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441364 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441306    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-ca-trust-extracted\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441364 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441354    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-registry-tls\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.441703 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.441677    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-ca-trust-extracted\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.442019 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.442001    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-registry-certificates\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.442197 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.442179    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-trusted-ca\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.443514 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.443496    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-image-registry-private-configuration\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.444006 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.443987    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-installation-pull-secrets\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.444105 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.444090    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-registry-tls\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.449659 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.449641    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-bound-sa-token\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.449872 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.449856    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5hh\" (UniqueName: \"kubernetes.io/projected/db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a-kube-api-access-8s5hh\") pod \"image-registry-749b8b8858-dc7d7\" (UID: \"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a\") " pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.612913 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.612849    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:25.882361 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.882252    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-749b8b8858-dc7d7"]
Sep 09 20:53:25.885439 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:25.885409    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb52dd3a_2f1a_4a0a_bfe4_1cb544b4c53a.slice/crio-64a975484f57f572166bbe9f28f2f57f00b5dfa5e81e23e8520046558b64a9be WatchSource:0}: Error finding container 64a975484f57f572166bbe9f28f2f57f00b5dfa5e81e23e8520046558b64a9be: Status 404 returned error can't find the container with id 64a975484f57f572166bbe9f28f2f57f00b5dfa5e81e23e8520046558b64a9be
Sep 09 20:53:25.894705 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:25.894675    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7" event={"ID":"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a","Type":"ContainerStarted","Data":"64a975484f57f572166bbe9f28f2f57f00b5dfa5e81e23e8520046558b64a9be"}
Sep 09 20:53:25.963837 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:25.963799    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5hfr5" podUID="870cf306-82d5-4437-9477-5c5d924506e8"
Sep 09 20:53:26.898980 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:26.898944    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7" event={"ID":"db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a","Type":"ContainerStarted","Data":"51e93df90f5bd0c401231fc4e3dcd791b6a882e503ee48004815e1b7fa8e620a"}
Sep 09 20:53:26.899354 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:26.899066    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:26.927165 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:26.927119    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7" podStartSLOduration=1.9271065429999998 podStartE2EDuration="1.927106543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:26.926808475 +0000 UTC m=+295.429638476" watchObservedRunningTime="2026-09-09 20:53:26.927106543 +0000 UTC m=+295.429936544"
Sep 09 20:53:26.952133 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:26.952107    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:53:26.954554 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:26.954531    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d6d21b7-47d9-423f-9127-f52cec9a1903-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-49fbc\" (UID: \"1d6d21b7-47d9-423f-9127-f52cec9a1903\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:53:27.253392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:27.253358    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"
Sep 09 20:53:27.526304 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:27.526243    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc"]
Sep 09 20:53:27.529279 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:27.529254    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6d21b7_47d9_423f_9127_f52cec9a1903.slice/crio-d580afbb89688bbc03a7deaa808cf24bc6710646f39f6a46b53ea488635c84a9 WatchSource:0}: Error finding container d580afbb89688bbc03a7deaa808cf24bc6710646f39f6a46b53ea488635c84a9: Status 404 returned error can't find the container with id d580afbb89688bbc03a7deaa808cf24bc6710646f39f6a46b53ea488635c84a9
Sep 09 20:53:27.868947 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:27.868874    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wl49r"
Sep 09 20:53:27.903207 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:27.903174    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc" event={"ID":"1d6d21b7-47d9-423f-9127-f52cec9a1903","Type":"ContainerStarted","Data":"d580afbb89688bbc03a7deaa808cf24bc6710646f39f6a46b53ea488635c84a9"}
Sep 09 20:53:29.915452 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:29.915411    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc" event={"ID":"1d6d21b7-47d9-423f-9127-f52cec9a1903","Type":"ContainerStarted","Data":"d30a5aafa5cb06710ba5b91858a1db4064861645c7d58743dbf61873101c6fd3"}
Sep 09 20:53:29.931127 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:29.931079    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-49fbc" podStartSLOduration=33.286956263 podStartE2EDuration="34.931067264s" totalImagesPullingTime="1.644111001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:29.929714947 +0000 UTC m=+298.432544949" watchObservedRunningTime="2026-09-09 20:53:29.931067264 +0000 UTC m=+298.433897268"
Sep 09 20:53:31.916654 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:31.916621    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:53:31.917099 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:31.916677    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:53:31.927564 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:31.927540    2548 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:53:35.228273 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:35.228239    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:37.962838 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:37.962804    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:53:38.081495 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.081467    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-txtd2"]
Sep 09 20:53:38.084624 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.084606    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.090859 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.090830    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:38.090975 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.090899    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:38.090975 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.090922    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:38.090975 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.090931    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2sv5v\""
Sep 09 20:53:38.091179 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.091161    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:38.131122 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131099    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-textfile\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131210 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131131    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635c8682-064a-424e-b68f-e7afac927bec-metrics-client-ca\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131210 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131151    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-root\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131210 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131175    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-sys\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131234    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-accelerators-collector-config\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131255    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131272    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-wtmp\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131290    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.131461 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.131312    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w96m\" (UniqueName: \"kubernetes.io/projected/635c8682-064a-424e-b68f-e7afac927bec-kube-api-access-9w96m\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232521 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232469    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-accelerators-collector-config\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232521 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232497    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232521 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232514    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-wtmp\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232701 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232537    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232701 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232561    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w96m\" (UniqueName: \"kubernetes.io/projected/635c8682-064a-424e-b68f-e7afac927bec-kube-api-access-9w96m\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232701 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232612    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-textfile\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232701 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:38.232640    2548 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:38.232701 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232686    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-wtmp\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232965 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232640    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635c8682-064a-424e-b68f-e7afac927bec-metrics-client-ca\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232965 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:38.232724    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls podName:635c8682-064a-424e-b68f-e7afac927bec nodeName:}" failed. No retries permitted until 2026-09-09 20:53:38.732704694 +0000 UTC m=+307.235534678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls") pod "node-exporter-txtd2" (UID: "635c8682-064a-424e-b68f-e7afac927bec") : secret "node-exporter-tls" not found
Sep 09 20:53:38.232965 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232796    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-root\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232965 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232839    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-sys\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232965 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232893    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-root\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.232965 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.232914    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/635c8682-064a-424e-b68f-e7afac927bec-sys\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.233184 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.233090    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-textfile\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.233184 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.233108    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-accelerators-collector-config\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.233184 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.233111    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635c8682-064a-424e-b68f-e7afac927bec-metrics-client-ca\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.234856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.234838    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.243696 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.243676    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w96m\" (UniqueName: \"kubernetes.io/projected/635c8682-064a-424e-b68f-e7afac927bec-kube-api-access-9w96m\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.736409 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:38.736378    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:38.736567 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:38.736526    2548 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:38.736607 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:38.736583    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls podName:635c8682-064a-424e-b68f-e7afac927bec nodeName:}" failed. No retries permitted until 2026-09-09 20:53:39.736566598 +0000 UTC m=+308.239396585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls") pod "node-exporter-txtd2" (UID: "635c8682-064a-424e-b68f-e7afac927bec") : secret "node-exporter-tls" not found
Sep 09 20:53:39.745941 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:39.745901    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:39.748277 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:39.748251    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/635c8682-064a-424e-b68f-e7afac927bec-node-exporter-tls\") pod \"node-exporter-txtd2\" (UID: \"635c8682-064a-424e-b68f-e7afac927bec\") " pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:39.893878 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:39.893843    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-txtd2"
Sep 09 20:53:39.905017 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:39.904983    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635c8682_064a_424e_b68f_e7afac927bec.slice/crio-f18cdfddaf76acbbd04fff4c950c7cae6dbaed98ad576362df43741555845f97 WatchSource:0}: Error finding container f18cdfddaf76acbbd04fff4c950c7cae6dbaed98ad576362df43741555845f97: Status 404 returned error can't find the container with id f18cdfddaf76acbbd04fff4c950c7cae6dbaed98ad576362df43741555845f97
Sep 09 20:53:39.906961 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:39.906938    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:53:39.946982 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:39.946956    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txtd2" event={"ID":"635c8682-064a-424e-b68f-e7afac927bec","Type":"ContainerStarted","Data":"f18cdfddaf76acbbd04fff4c950c7cae6dbaed98ad576362df43741555845f97"}
Sep 09 20:53:40.952293 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:40.952254    2548 generic.go:356] "Generic (PLEG): container finished" podID="635c8682-064a-424e-b68f-e7afac927bec" containerID="bf7d82ce37b9d8e1b8691097c4b8c24015a49c6adc3386b53704af7b470f258f" exitCode=0
Sep 09 20:53:40.952293 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:40.952297    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txtd2" event={"ID":"635c8682-064a-424e-b68f-e7afac927bec","Type":"ContainerDied","Data":"bf7d82ce37b9d8e1b8691097c4b8c24015a49c6adc3386b53704af7b470f258f"}
Sep 09 20:53:41.959161 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:41.959128    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txtd2" event={"ID":"635c8682-064a-424e-b68f-e7afac927bec","Type":"ContainerStarted","Data":"e0140cf29afb42b824fde44e596ec3fd38e80fe624c918153cdf63988c40f7be"}
Sep 09 20:53:41.959526 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:41.959182    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txtd2" event={"ID":"635c8682-064a-424e-b68f-e7afac927bec","Type":"ContainerStarted","Data":"46bf2ff03fa5eed569175d8bf7679f5b5fc94a1c699eb6c47d4c0e35cb4303bf"}
Sep 09 20:53:42.004917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:42.004863    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-txtd2" podStartSLOduration=3.112662588 podStartE2EDuration="4.004848219s" totalImagesPullingTime="827.535824ms" totalInitContainerRuntime="64.649807ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:42.004089616 +0000 UTC m=+310.506919618" watchObservedRunningTime="2026-09-09 20:53:42.004848219 +0000 UTC m=+310.507678219"
Sep 09 20:53:44.461974 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.461939    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:53:44.465658 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.465638    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.472079 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.471752    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 20:53:44.472079 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.471777    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 20:53:44.473984 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.473963    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 20:53:44.474126 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.474107    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 20:53:44.474235 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.474219    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 20:53:44.474388 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.474373    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 20:53:44.474462 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.474427    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 20:53:44.474516 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.474498    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4gnvl\""
Sep 09 20:53:44.475117 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475103    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 20:53:44.475198 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475121    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 20:53:44.475198 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475128    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 20:53:44.475198 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475141    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 20:53:44.475198 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475128    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 20:53:44.475352 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475121    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 20:53:44.475352 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.475336    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2e455sc3b6uq9\""
Sep 09 20:53:44.477107 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.477090    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 20:53:44.481588 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.481573    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 20:53:44.495968 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.495945    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:53:44.585580 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585545    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.585775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585585    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.585775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585624    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.585775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585648    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.585775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585670    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.585775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585693    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585804    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585832    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585856    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585930    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585959    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.585980    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586001    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586028    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586052    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jn5v\" (UniqueName: \"kubernetes.io/projected/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-kube-api-access-7jn5v\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586077    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586093    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586128    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-config\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586167    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.586244 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.586189    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687516 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687480    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687516 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687514    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687531    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687551    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687573    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687591    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687607    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687635    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.687758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687651    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687762    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687814    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687844    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687876    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jn5v\" (UniqueName: \"kubernetes.io/projected/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-kube-api-access-7jn5v\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687912    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687935    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.687963    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-config\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.688018    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.688055    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.688089    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688569 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.688123    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.688569 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.688476    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.689055 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.689026    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.689879 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.689649    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.691233 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.690842    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.691233 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.690929    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.691233 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.690932    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.691434 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.691302    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.691434 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.691344    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.692363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.691837    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.692363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.691929    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.692363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.692102    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.692575 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.692376    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.692633 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.692576    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.692918 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.692897    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.693010 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.692920    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.693110 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.693090    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.693346 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.693327    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.693711 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.693687    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.694333 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.694316    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-config\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.702428 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.702407    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jn5v\" (UniqueName: \"kubernetes.io/projected/b8bcd4a3-3b74-44c0-8d92-6e00fc010637-kube-api-access-7jn5v\") pod \"prometheus-k8s-0\" (UID: \"b8bcd4a3-3b74-44c0-8d92-6e00fc010637\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:44.774576 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:44.774507    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:53:45.057242 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:45.057151    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 20:53:45.060821 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:53:45.060794    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bcd4a3_3b74_44c0_8d92_6e00fc010637.slice/crio-cdfacfd74cfd42292515236454c5de96c191b2fdc5cf7b7bb6158a68c02e10ef WatchSource:0}: Error finding container cdfacfd74cfd42292515236454c5de96c191b2fdc5cf7b7bb6158a68c02e10ef: Status 404 returned error can't find the container with id cdfacfd74cfd42292515236454c5de96c191b2fdc5cf7b7bb6158a68c02e10ef
Sep 09 20:53:45.974673 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:45.974632    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"cdfacfd74cfd42292515236454c5de96c191b2fdc5cf7b7bb6158a68c02e10ef"}
Sep 09 20:53:46.979055 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:46.979015    2548 generic.go:356] "Generic (PLEG): container finished" podID="b8bcd4a3-3b74-44c0-8d92-6e00fc010637" containerID="2ebd04db0816a6f6b09acd17b6610921b06d2b86cc0c4141b0b09ee97b9175a8" exitCode=0
Sep 09 20:53:46.979486 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:46.979089    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerDied","Data":"2ebd04db0816a6f6b09acd17b6610921b06d2b86cc0c4141b0b09ee97b9175a8"}
Sep 09 20:53:47.908522 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:47.908487    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-749b8b8858-dc7d7"
Sep 09 20:53:50.236774 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.236686    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8" podUID="be85228e-37c3-4a62-8108-7068b40b7881" containerName="registry" containerID="cri-o://8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3" gracePeriod=30
Sep 09 20:53:50.494200 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.494147    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:50.646279 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646242    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5pm2\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-kube-api-access-l5pm2\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646279 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646289    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-registry-certificates\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646535 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646345    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be85228e-37c3-4a62-8108-7068b40b7881-ca-trust-extracted\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646535 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646426    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646535 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646463    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-trusted-ca\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646535 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646503    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-bound-sa-token\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646535 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646527    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-image-registry-private-configuration\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646872 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646572    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-installation-pull-secrets\") pod \"be85228e-37c3-4a62-8108-7068b40b7881\" (UID: \"be85228e-37c3-4a62-8108-7068b40b7881\") "
Sep 09 20:53:50.646872 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646778    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:50.646972 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646888    2548 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-registry-certificates\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.646972 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.646886    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:50.648996 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.648947    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:50.648996 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.648970    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-kube-api-access-l5pm2" (OuterVolumeSpecName: "kube-api-access-l5pm2") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "kube-api-access-l5pm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:50.649167 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.649153    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:50.649227 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.649203    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:50.649467 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.649441    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:50.655845 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.655822    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be85228e-37c3-4a62-8108-7068b40b7881-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "be85228e-37c3-4a62-8108-7068b40b7881" (UID: "be85228e-37c3-4a62-8108-7068b40b7881"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.747997    2548 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be85228e-37c3-4a62-8108-7068b40b7881-ca-trust-extracted\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.748026    2548 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-registry-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.748038    2548 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be85228e-37c3-4a62-8108-7068b40b7881-trusted-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.748046    2548 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-bound-sa-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.748054    2548 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-image-registry-private-configuration\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.748064    2548 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be85228e-37c3-4a62-8108-7068b40b7881-installation-pull-secrets\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.748076 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.748073    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5pm2\" (UniqueName: \"kubernetes.io/projected/be85228e-37c3-4a62-8108-7068b40b7881-kube-api-access-l5pm2\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.995220 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.995187    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"42e1ed93ad52fa1107c38b14fe23176391bde8539dd03b27e0bbad9f5888fbfa"}
Sep 09 20:53:50.995393 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.995248    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"d478a25aacb4b1d1337c0b08d186dcf288b0899c05f96772133dfa1cc483d464"}
Sep 09 20:53:50.996398 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.996369    2548 generic.go:356] "Generic (PLEG): container finished" podID="be85228e-37c3-4a62-8108-7068b40b7881" containerID="8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3" exitCode=0
Sep 09 20:53:50.996507 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.996416    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"
Sep 09 20:53:50.996507 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.996444    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8" event={"ID":"be85228e-37c3-4a62-8108-7068b40b7881","Type":"ContainerDied","Data":"8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3"}
Sep 09 20:53:50.996507 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.996475    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8" event={"ID":"be85228e-37c3-4a62-8108-7068b40b7881","Type":"ContainerDied","Data":"ab43deba8ce5e05108e9760faf97ed8a64ec560bc19a72d4e6b0e7837ddf5077"}
Sep 09 20:53:50.996507 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:50.996493    2548 scope.go:117] "RemoveContainer" containerID="8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3"
Sep 09 20:53:51.004385 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:51.004362    2548 scope.go:117] "RemoveContainer" containerID="8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3"
Sep 09 20:53:51.004675 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:53:51.004648    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3\": container with ID starting with 8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3 not found: ID does not exist" containerID="8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3"
Sep 09 20:53:51.004770 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:51.004686    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3"} err="failed to get container status \"8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3\": rpc error: code = NotFound desc = could not find container \"8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3\": container with ID starting with 8f2451b46b9ff22bb097c00dace29b422723b31e863f1fbc9f98b8573dad2ee3 not found: ID does not exist"
Sep 09 20:53:51.016758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:51.016721    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"]
Sep 09 20:53:51.022958 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:51.022939    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bb6b9dcbd-bfqm8"]
Sep 09 20:53:51.968329 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:51.968301    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be85228e-37c3-4a62-8108-7068b40b7881" path="/var/lib/kubelet/pods/be85228e-37c3-4a62-8108-7068b40b7881/volumes"
Sep 09 20:53:53.008900 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:53.008862    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"62fc0beac763f100f8e410ee70f6003951da28d6526b9fd3487cec4d9c247325"}
Sep 09 20:53:53.009285 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:53.008981    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"fc804578c55c370cb028313f68e682e3b66312c27c9a57704ba16f54bcbc038b"}
Sep 09 20:53:53.009285 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:53.009023    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"9970b57d6968db9f9ff0a53118a21d92b19fe5c0ff9bb676a214f8a930f964b2"}
Sep 09 20:53:53.009285 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:53.009065    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8bcd4a3-3b74-44c0-8d92-6e00fc010637","Type":"ContainerStarted","Data":"e1dd07e367b62e86578ac5d7a952624923949c8790fcc31a4405bfcab37a0510"}
Sep 09 20:53:53.042008 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:53.041962    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.594448365 podStartE2EDuration="9.04194288s" totalImagesPullingTime="6.428536639s" totalInitContainerRuntime="18.957876ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:53.040283245 +0000 UTC m=+321.543113247" watchObservedRunningTime="2026-09-09 20:53:53.04194288 +0000 UTC m=+321.544772882"
Sep 09 20:53:54.774981 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:53:54.774947    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:54:44.774807 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:44.774768    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:54:44.789862 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:44.789833    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:54:45.195862 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:45.195834    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 20:54:45.765303 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:45.765274    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:54:45.767589 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:45.767557    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/870cf306-82d5-4437-9477-5c5d924506e8-metrics-certs\") pod \"network-metrics-daemon-5hfr5\" (UID: \"870cf306-82d5-4437-9477-5c5d924506e8\") " pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:54:46.066555 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:46.066475    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6qsr\""
Sep 09 20:54:46.073568 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:46.073550    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5hfr5"
Sep 09 20:54:46.343095 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:46.343071    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5hfr5"]
Sep 09 20:54:46.345446 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:54:46.345412    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870cf306_82d5_4437_9477_5c5d924506e8.slice/crio-14f4eafa9f3565958168d54a5219bf60ec4a629e751ef6ef733fa54f441b4778 WatchSource:0}: Error finding container 14f4eafa9f3565958168d54a5219bf60ec4a629e751ef6ef733fa54f441b4778: Status 404 returned error can't find the container with id 14f4eafa9f3565958168d54a5219bf60ec4a629e751ef6ef733fa54f441b4778
Sep 09 20:54:47.191089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:47.191044    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5hfr5" event={"ID":"870cf306-82d5-4437-9477-5c5d924506e8","Type":"ContainerStarted","Data":"14f4eafa9f3565958168d54a5219bf60ec4a629e751ef6ef733fa54f441b4778"}
Sep 09 20:54:48.196597 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:48.196564    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5hfr5" event={"ID":"870cf306-82d5-4437-9477-5c5d924506e8","Type":"ContainerStarted","Data":"d720f6d9b7de1b92da1cf49d7208dd8bddcf61182c1358d63decbb4973d67373"}
Sep 09 20:54:48.196965 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:48.196613    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5hfr5" event={"ID":"870cf306-82d5-4437-9477-5c5d924506e8","Type":"ContainerStarted","Data":"d57b21cbdb68983925c0ddee005615da74f9b71ab8e07afbc4fe59e776fa2dcf"}
Sep 09 20:54:48.223473 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:54:48.223423    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5hfr5" podStartSLOduration=375.315216659 podStartE2EDuration="6m16.223403844s" totalImagesPullingTime="908.187185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:48.222656687 +0000 UTC m=+376.725486687" watchObservedRunningTime="2026-09-09 20:54:48.223403844 +0000 UTC m=+376.726233847"
Sep 09 20:57:30.088276 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.088186    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"]
Sep 09 20:57:30.088785 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.088562    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be85228e-37c3-4a62-8108-7068b40b7881" containerName="registry"
Sep 09 20:57:30.088785 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.088581    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="be85228e-37c3-4a62-8108-7068b40b7881" containerName="registry"
Sep 09 20:57:30.088785 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.088646    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="be85228e-37c3-4a62-8108-7068b40b7881" containerName="registry"
Sep 09 20:57:30.091631 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.091609    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.094320 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.094301    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bkhht\""
Sep 09 20:57:30.094574 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.094553    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 20:57:30.094823 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.094807    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 20:57:30.094921 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.094808    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 20:57:30.100955 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.100938    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 20:57:30.105899 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.105881    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"]
Sep 09 20:57:30.112991 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.112971    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-webhook-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.113081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.113001    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbww\" (UniqueName: \"kubernetes.io/projected/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-kube-api-access-hwbww\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.113081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.113021    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-manifests\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.113081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.113051    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-tmp\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.113081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.113077    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.213486 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.213463    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-webhook-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.213615 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.213502    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbww\" (UniqueName: \"kubernetes.io/projected/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-kube-api-access-hwbww\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.213615 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.213525    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-manifests\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.213615 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.213551    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-tmp\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.213615 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.213573    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.214036 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.214014    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-manifests\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.215619 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.215592    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-tmp\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.215725 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.215642    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-webhook-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.215837 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.215816    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.223808 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.223781    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbww\" (UniqueName: \"kubernetes.io/projected/c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3-kube-api-access-hwbww\") pod \"opendatahub-operator-controller-manager-d75fb655b-dq7m8\" (UID: \"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3\") " pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.401558 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.401506    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:30.675448 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.675380    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"]
Sep 09 20:57:30.678948 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:57:30.678916    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ac2e0a_c621_49cc_a57b_8c07ec7d29f3.slice/crio-9203a11852b6f6fa2809e50a7a00df02151df327f8c94f4e37ba93282a44ed0d WatchSource:0}: Error finding container 9203a11852b6f6fa2809e50a7a00df02151df327f8c94f4e37ba93282a44ed0d: Status 404 returned error can't find the container with id 9203a11852b6f6fa2809e50a7a00df02151df327f8c94f4e37ba93282a44ed0d
Sep 09 20:57:30.758880 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:30.758849    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8" event={"ID":"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3","Type":"ContainerStarted","Data":"9203a11852b6f6fa2809e50a7a00df02151df327f8c94f4e37ba93282a44ed0d"}
Sep 09 20:57:34.776299 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:34.776250    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8" event={"ID":"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3","Type":"ContainerStarted","Data":"46d6b73b02bde322752fcf20c953a3996704ae57b5abdb3869bbe19a502d7614"}
Sep 09 20:57:35.781670 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:35.781632    2548 generic.go:356] "Generic (PLEG): container finished" podID="c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3" containerID="46d6b73b02bde322752fcf20c953a3996704ae57b5abdb3869bbe19a502d7614" exitCode=0
Sep 09 20:57:35.782055 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:35.781685    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8" event={"ID":"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3","Type":"ContainerDied","Data":"46d6b73b02bde322752fcf20c953a3996704ae57b5abdb3869bbe19a502d7614"}
Sep 09 20:57:36.787800 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:36.787762    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8" event={"ID":"c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3","Type":"ContainerStarted","Data":"cecdbb6987a871a72a0f09e0b6efece9123e95afa1754023f91ae6d4f17658fd"}
Sep 09 20:57:36.788172 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:36.787914    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:57:36.808716 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:36.808672    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8" podStartSLOduration=2.030262318 podStartE2EDuration="6.808658603s" totalImagesPullingTime="3.765526884s" totalInitContainerRuntime="1.012869401s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:36.806825876 +0000 UTC m=+545.309655878" watchObservedRunningTime="2026-09-09 20:57:36.808658603 +0000 UTC m=+545.311488603"
Sep 09 20:57:47.794382 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:57:47.794346    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d75fb655b-dq7m8"
Sep 09 20:58:01.189127 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.189096    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"]
Sep 09 20:58:01.192172 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.192155    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.195765 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.195723    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 20:58:01.195886 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.195793    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 20:58:01.196375 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.196362    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sgvw8\""
Sep 09 20:58:01.239755 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.239710    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/497cefe0-cfdd-46c6-9585-018befe67067-tmp\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.239898 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.239810    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgrc\" (UniqueName: \"kubernetes.io/projected/497cefe0-cfdd-46c6-9585-018befe67067-kube-api-access-4mgrc\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.239898 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.239864    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497cefe0-cfdd-46c6-9585-018befe67067-tls-certs\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.277131 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.277103    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"]
Sep 09 20:58:01.341016 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.340982    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgrc\" (UniqueName: \"kubernetes.io/projected/497cefe0-cfdd-46c6-9585-018befe67067-kube-api-access-4mgrc\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.341192 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.341033    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497cefe0-cfdd-46c6-9585-018befe67067-tls-certs\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.341192 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.341080    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/497cefe0-cfdd-46c6-9585-018befe67067-tmp\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.343450 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.343421    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/497cefe0-cfdd-46c6-9585-018befe67067-tmp\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.343678 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.343655    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/497cefe0-cfdd-46c6-9585-018befe67067-tls-certs\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.370289 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.370250    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dgmjp"]
Sep 09 20:58:01.374351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.374329    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:01.379887 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.379863    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:58:01.380760 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.380720    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgrc\" (UniqueName: \"kubernetes.io/projected/497cefe0-cfdd-46c6-9585-018befe67067-kube-api-access-4mgrc\") pod \"kube-auth-proxy-6bcf9fffd5-g8tlr\" (UID: \"497cefe0-cfdd-46c6-9585-018befe67067\") " pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.380873 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.380843    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8295b\""
Sep 09 20:58:01.442066 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.441984    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnh2n\" (UniqueName: \"kubernetes.io/projected/d8cd73f5-2805-43a9-990c-32f7192d450d-kube-api-access-mnh2n\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:01.442066 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.442034    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8cd73f5-2805-43a9-990c-32f7192d450d-cert\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:01.487454 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.487425    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dgmjp"]
Sep 09 20:58:01.500974 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.500944    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"
Sep 09 20:58:01.543543 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.543507    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnh2n\" (UniqueName: \"kubernetes.io/projected/d8cd73f5-2805-43a9-990c-32f7192d450d-kube-api-access-mnh2n\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:01.543708 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.543568    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8cd73f5-2805-43a9-990c-32f7192d450d-cert\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:01.543807 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:01.543779    2548 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 20:58:01.543922 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:01.543866    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8cd73f5-2805-43a9-990c-32f7192d450d-cert podName:d8cd73f5-2805-43a9-990c-32f7192d450d nodeName:}" failed. No retries permitted until 2026-09-09 20:58:02.043842273 +0000 UTC m=+570.546672269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d8cd73f5-2805-43a9-990c-32f7192d450d-cert") pod "odh-model-controller-7998c67dc4-dgmjp" (UID: "d8cd73f5-2805-43a9-990c-32f7192d450d") : secret "odh-model-controller-webhook-cert" not found
Sep 09 20:58:01.557589 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.557564    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnh2n\" (UniqueName: \"kubernetes.io/projected/d8cd73f5-2805-43a9-990c-32f7192d450d-kube-api-access-mnh2n\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:01.792988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.792905    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr"]
Sep 09 20:58:01.796095 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:01.796067    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497cefe0_cfdd_46c6_9585_018befe67067.slice/crio-5f002ce2357a65ab05c9970df7b3c1dc07736c1086b4c39db356bee6ac8021a3 WatchSource:0}: Error finding container 5f002ce2357a65ab05c9970df7b3c1dc07736c1086b4c39db356bee6ac8021a3: Status 404 returned error can't find the container with id 5f002ce2357a65ab05c9970df7b3c1dc07736c1086b4c39db356bee6ac8021a3
Sep 09 20:58:01.879387 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:01.879353    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr" event={"ID":"497cefe0-cfdd-46c6-9585-018befe67067","Type":"ContainerStarted","Data":"5f002ce2357a65ab05c9970df7b3c1dc07736c1086b4c39db356bee6ac8021a3"}
Sep 09 20:58:02.049234 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.049153    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8cd73f5-2805-43a9-990c-32f7192d450d-cert\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:02.051574 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.051549    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8cd73f5-2805-43a9-990c-32f7192d450d-cert\") pod \"odh-model-controller-7998c67dc4-dgmjp\" (UID: \"d8cd73f5-2805-43a9-990c-32f7192d450d\") " pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:02.219347 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.219315    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ssnpl"]
Sep 09 20:58:02.223473 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.223457    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.228070 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.228051    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 20:58:02.228175 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.228117    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-97jdp\""
Sep 09 20:58:02.236606 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.236584    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ssnpl"]
Sep 09 20:58:02.289396 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.289365    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:02.352358 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.352325    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h99zd\" (UniqueName: \"kubernetes.io/projected/34334705-4ee6-4798-b603-cbd03350e9c9-kube-api-access-h99zd\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.352471 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.352412    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.453940 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.453900    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.454135 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.454002    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h99zd\" (UniqueName: \"kubernetes.io/projected/34334705-4ee6-4798-b603-cbd03350e9c9-kube-api-access-h99zd\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.454209 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:02.454149    2548 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:58:02.454261 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:02.454242    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs podName:34334705-4ee6-4798-b603-cbd03350e9c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:58:02.954217596 +0000 UTC m=+571.457047577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs") pod "model-serving-api-54d89bd5fc-ssnpl" (UID: "34334705-4ee6-4798-b603-cbd03350e9c9") : secret "model-serving-api-tls" not found
Sep 09 20:58:02.467752 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.467699    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h99zd\" (UniqueName: \"kubernetes.io/projected/34334705-4ee6-4798-b603-cbd03350e9c9-kube-api-access-h99zd\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.569839 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.569688    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dgmjp"]
Sep 09 20:58:02.573266 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:02.573236    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cd73f5_2805_43a9_990c_32f7192d450d.slice/crio-9bde292f54790c0715797c7984561c77d43f3cb42cd7945a86b4d2959efca1ce WatchSource:0}: Error finding container 9bde292f54790c0715797c7984561c77d43f3cb42cd7945a86b4d2959efca1ce: Status 404 returned error can't find the container with id 9bde292f54790c0715797c7984561c77d43f3cb42cd7945a86b4d2959efca1ce
Sep 09 20:58:02.885457 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.885352    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp" event={"ID":"d8cd73f5-2805-43a9-990c-32f7192d450d","Type":"ContainerStarted","Data":"9bde292f54790c0715797c7984561c77d43f3cb42cd7945a86b4d2959efca1ce"}
Sep 09 20:58:02.917159 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.917118    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"]
Sep 09 20:58:02.920906 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.920879    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:02.923571 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.923548    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:58:02.924115 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.924094    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fgjvh\""
Sep 09 20:58:02.924751 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.924718    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:58:02.934418 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.934401    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:58:02.935553 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.935531    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:58:02.948217 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.948193    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"]
Sep 09 20:58:02.958397 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:02.958321    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:02.958833 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:02.958498    2548 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:58:02.958833 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:02.958578    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs podName:34334705-4ee6-4798-b603-cbd03350e9c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:58:03.958556825 +0000 UTC m=+572.461386806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs") pod "model-serving-api-54d89bd5fc-ssnpl" (UID: "34334705-4ee6-4798-b603-cbd03350e9c9") : secret "model-serving-api-tls" not found
Sep 09 20:58:03.058987 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.058946    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.059175 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.059009    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.059175 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.059041    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.059175 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.059081    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.059175 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.059120    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.059175 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.059157    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.059406 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.059225    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46x6\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-kube-api-access-j46x6\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160292 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160193    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160292 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160246    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160292 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160291    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160548 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160320    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160548 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160358    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160548 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160410    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46x6\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-kube-api-access-j46x6\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.160548 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.160467    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.161381 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.161317    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.163528 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.163492    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.163805 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.163778    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.163919 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.163822    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.163919 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.163866    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.176540 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.176458    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46x6\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-kube-api-access-j46x6\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.180184 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.180154    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-k76qg\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.234183 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.234132    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:03.968191 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.968079    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:03.971799 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:03.971766    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34334705-4ee6-4798-b603-cbd03350e9c9-tls-certs\") pod \"model-serving-api-54d89bd5fc-ssnpl\" (UID: \"34334705-4ee6-4798-b603-cbd03350e9c9\") " pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:04.033793 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:04.033717    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:04.896885 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:04.896849    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr" event={"ID":"497cefe0-cfdd-46c6-9585-018befe67067","Type":"ContainerStarted","Data":"7d2fc37a655d246cfc6c934607d98dffe2b316a7d3e24ce3a5ca7c01c5cd7cfa"}
Sep 09 20:58:04.928758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:04.928692    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bcf9fffd5-g8tlr" podStartSLOduration=0.970972403 podStartE2EDuration="3.928674028s" totalImagesPullingTime="2.957701625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:04.9274796 +0000 UTC m=+573.430309603" watchObservedRunningTime="2026-09-09 20:58:04.928674028 +0000 UTC m=+573.431504028"
Sep 09 20:58:04.990546 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:04.990506    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ssnpl"]
Sep 09 20:58:05.000026 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:05.000001    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"]
Sep 09 20:58:05.537310 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:05.537274    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34334705_4ee6_4798_b603_cbd03350e9c9.slice/crio-fb5c7772d6ef884bf3b472b6ff88e572b60be92df4ba66f51314cd7a21a68ef4 WatchSource:0}: Error finding container fb5c7772d6ef884bf3b472b6ff88e572b60be92df4ba66f51314cd7a21a68ef4: Status 404 returned error can't find the container with id fb5c7772d6ef884bf3b472b6ff88e572b60be92df4ba66f51314cd7a21a68ef4
Sep 09 20:58:05.903276 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:05.903182    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl" event={"ID":"34334705-4ee6-4798-b603-cbd03350e9c9","Type":"ContainerStarted","Data":"fb5c7772d6ef884bf3b472b6ff88e572b60be92df4ba66f51314cd7a21a68ef4"}
Sep 09 20:58:06.064700 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:06.064667    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c174c7e_ab77_4cc9_b0e2_5156e1eed257.slice/crio-515e0fc5630a7738b10a0f4375724b6c9c1600f04d7eb798224267cc442f1351 WatchSource:0}: Error finding container 515e0fc5630a7738b10a0f4375724b6c9c1600f04d7eb798224267cc442f1351: Status 404 returned error can't find the container with id 515e0fc5630a7738b10a0f4375724b6c9c1600f04d7eb798224267cc442f1351
Sep 09 20:58:06.909821 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:06.909774    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" event={"ID":"7c174c7e-ab77-4cc9-b0e2-5156e1eed257","Type":"ContainerStarted","Data":"515e0fc5630a7738b10a0f4375724b6c9c1600f04d7eb798224267cc442f1351"}
Sep 09 20:58:06.911560 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:06.911524    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp" event={"ID":"d8cd73f5-2805-43a9-990c-32f7192d450d","Type":"ContainerStarted","Data":"847430cfe8c7ab2f17ae4c03e8b6bb8b0fe68fd88e3a0082f43b2bb2190196ee"}
Sep 09 20:58:06.911691 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:06.911645    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:06.937483 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:06.937428    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp" podStartSLOduration=2.398030393 podStartE2EDuration="5.937408995s" totalImagesPullingTime="3.539378602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:06.937219505 +0000 UTC m=+575.440049507" watchObservedRunningTime="2026-09-09 20:58:06.937408995 +0000 UTC m=+575.440238998"
Sep 09 20:58:07.917588 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:07.917551    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl" event={"ID":"34334705-4ee6-4798-b603-cbd03350e9c9","Type":"ContainerStarted","Data":"fab46d6b7d55e1175c1e7f2d84979dc03e27ebf7238e55743e0208f9d3aeecda"}
Sep 09 20:58:07.918081 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:07.917694    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:07.944273 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:07.944215    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl" podStartSLOduration=4.329924158 podStartE2EDuration="5.944196526s" totalImagesPullingTime="1.614272368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:07.942271103 +0000 UTC m=+576.445101104" watchObservedRunningTime="2026-09-09 20:58:07.944196526 +0000 UTC m=+576.447026528"
Sep 09 20:58:08.911172 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:08.911135    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:08.911272 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:08.911202    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:09.928381 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:09.928332    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" event={"ID":"7c174c7e-ab77-4cc9-b0e2-5156e1eed257","Type":"ContainerStarted","Data":"8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2"}
Sep 09 20:58:09.928789 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:09.928510    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:09.929988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:09.929959    2548 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-k76qg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:58:09.930109 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:09.930006    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" podUID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:58:09.976720 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:09.976677    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" podStartSLOduration=5.132485364 podStartE2EDuration="7.9766648s" totalImagesPullingTime="2.844179436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:09.974102098 +0000 UTC m=+578.476932098" watchObservedRunningTime="2026-09-09 20:58:09.9766648 +0000 UTC m=+578.479494801"
Sep 09 20:58:10.359043 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.359013    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"]
Sep 09 20:58:10.362349 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.362327    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.367660 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.367638    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ns67d\""
Sep 09 20:58:10.387504 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.387479    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"]
Sep 09 20:58:10.427232 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427202    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427319 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427250    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5917a147-4b51-484b-9b23-59733727a52e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427381 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427350    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtt9z\" (UniqueName: \"kubernetes.io/projected/5917a147-4b51-484b-9b23-59733727a52e-kube-api-access-jtt9z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427423 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427394    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5917a147-4b51-484b-9b23-59733727a52e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427465 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427450    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427507 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427466    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427544 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427500    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427544 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427528    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5917a147-4b51-484b-9b23-59733727a52e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.427608 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.427590    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528185 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528157    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtt9z\" (UniqueName: \"kubernetes.io/projected/5917a147-4b51-484b-9b23-59733727a52e-kube-api-access-jtt9z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528307 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528197    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5917a147-4b51-484b-9b23-59733727a52e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528307 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528244    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528307 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528263    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528307 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528277    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528307 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528297    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5917a147-4b51-484b-9b23-59733727a52e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528545 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528324    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528545 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528369    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528545 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528428    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5917a147-4b51-484b-9b23-59733727a52e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528704 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528677    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528868 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528846    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528909 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528874    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.528942 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.528919    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.529245 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.529228    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5917a147-4b51-484b-9b23-59733727a52e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.530620 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.530599    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5917a147-4b51-484b-9b23-59733727a52e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.530840 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.530822    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5917a147-4b51-484b-9b23-59733727a52e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.540885 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.540862    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5917a147-4b51-484b-9b23-59733727a52e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.540959 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.540891    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtt9z\" (UniqueName: \"kubernetes.io/projected/5917a147-4b51-484b-9b23-59733727a52e-kube-api-access-jtt9z\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk\" (UID: \"5917a147-4b51-484b-9b23-59733727a52e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.672349 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.672279    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:10.932713 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.932616    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:10.955998 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:10.955856    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"]
Sep 09 20:58:10.958470 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:10.958442    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5917a147_4b51_484b_9b23_59733727a52e.slice/crio-38acdc6f82d02c755de8c5be5369a93f29b03d64d21ebe885b0bed205cf918c0 WatchSource:0}: Error finding container 38acdc6f82d02c755de8c5be5369a93f29b03d64d21ebe885b0bed205cf918c0: Status 404 returned error can't find the container with id 38acdc6f82d02c755de8c5be5369a93f29b03d64d21ebe885b0bed205cf918c0
Sep 09 20:58:11.938149 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:11.938114    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk" event={"ID":"5917a147-4b51-484b-9b23-59733727a52e","Type":"ContainerStarted","Data":"38acdc6f82d02c755de8c5be5369a93f29b03d64d21ebe885b0bed205cf918c0"}
Sep 09 20:58:13.535664 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:13.535626    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:13.535917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:13.535707    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:13.535917 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:13.535762    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:58:13.948547 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:13.948508    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk" event={"ID":"5917a147-4b51-484b-9b23-59733727a52e","Type":"ContainerStarted","Data":"9b587939b3b0e813acf102c1b5b6c08bb871c353a4756751e977bbe3f684a6e0"}
Sep 09 20:58:14.117301 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.117249    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk" podStartSLOduration=1.542795308 podStartE2EDuration="4.117232351s" totalImagesPullingTime="2.574437043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:14.108963239 +0000 UTC m=+582.611793242" watchObservedRunningTime="2026-09-09 20:58:14.117232351 +0000 UTC m=+582.620062351"
Sep 09 20:58:14.673232 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.673202    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:14.677571 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.677547    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:14.949220 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.949188    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"]
Sep 09 20:58:14.952615 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.952594    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:14.954197 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.954162    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:14.954994 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:14.954979    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk"
Sep 09 20:58:14.955859 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:14.955835    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret"
Sep 09 20:58:14.955934 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:14.955837    2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-controller-manager-dockercfg-4x2c5\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4x2c5\"" type="*v1.Secret"
Sep 09 20:58:15.014970 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.014942    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"]
Sep 09 20:58:15.040178 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.040155    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"]
Sep 09 20:58:15.043398 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.043382    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.045877 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.045861    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:58:15.045993 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.045974    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ch5bw\""
Sep 09 20:58:15.066122 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.066103    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4751359f-6f18-4dca-ab21-69d8583c0bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-smxn7\" (UID: \"4751359f-6f18-4dca-ab21-69d8583c0bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:15.066266 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.066246    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshwl\" (UniqueName: \"kubernetes.io/projected/4751359f-6f18-4dca-ab21-69d8583c0bc3-kube-api-access-jshwl\") pod \"kserve-controller-manager-d6f4b65b8-smxn7\" (UID: \"4751359f-6f18-4dca-ab21-69d8583c0bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:15.073978 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.073951    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"]
Sep 09 20:58:15.166813 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.166780    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshwl\" (UniqueName: \"kubernetes.io/projected/4751359f-6f18-4dca-ab21-69d8583c0bc3-kube-api-access-jshwl\") pod \"kserve-controller-manager-d6f4b65b8-smxn7\" (UID: \"4751359f-6f18-4dca-ab21-69d8583c0bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:15.166939 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.166819    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxptx\" (UniqueName: \"kubernetes.io/projected/e6679e73-069b-424d-9cce-972deda45aa6-kube-api-access-bxptx\") pod \"llmisvc-controller-manager-777c6bd7d5-cd2bt\" (UID: \"e6679e73-069b-424d-9cce-972deda45aa6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.166939 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.166914    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4751359f-6f18-4dca-ab21-69d8583c0bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-smxn7\" (UID: \"4751359f-6f18-4dca-ab21-69d8583c0bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:15.167026 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.166974    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6679e73-069b-424d-9cce-972deda45aa6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cd2bt\" (UID: \"e6679e73-069b-424d-9cce-972deda45aa6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.189595 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.189565    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshwl\" (UniqueName: \"kubernetes.io/projected/4751359f-6f18-4dca-ab21-69d8583c0bc3-kube-api-access-jshwl\") pod \"kserve-controller-manager-d6f4b65b8-smxn7\" (UID: \"4751359f-6f18-4dca-ab21-69d8583c0bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:15.267795 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.267715    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxptx\" (UniqueName: \"kubernetes.io/projected/e6679e73-069b-424d-9cce-972deda45aa6-kube-api-access-bxptx\") pod \"llmisvc-controller-manager-777c6bd7d5-cd2bt\" (UID: \"e6679e73-069b-424d-9cce-972deda45aa6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.267950 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.267827    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6679e73-069b-424d-9cce-972deda45aa6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cd2bt\" (UID: \"e6679e73-069b-424d-9cce-972deda45aa6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.270018 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.269999    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6679e73-069b-424d-9cce-972deda45aa6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cd2bt\" (UID: \"e6679e73-069b-424d-9cce-972deda45aa6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.285914 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.285887    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxptx\" (UniqueName: \"kubernetes.io/projected/e6679e73-069b-424d-9cce-972deda45aa6-kube-api-access-bxptx\") pod \"llmisvc-controller-manager-777c6bd7d5-cd2bt\" (UID: \"e6679e73-069b-424d-9cce-972deda45aa6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.352137 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.352111    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:15.689024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.688992    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"]
Sep 09 20:58:15.699317 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:15.699295    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode6679e73_069b_424d_9cce_972deda45aa6.slice/crio-937ad6bf44b6827466d0462eb72cf77498193696844a842b46578cb266b6b24c WatchSource:0}: Error finding container 937ad6bf44b6827466d0462eb72cf77498193696844a842b46578cb266b6b24c: Status 404 returned error can't find the container with id 937ad6bf44b6827466d0462eb72cf77498193696844a842b46578cb266b6b24c
Sep 09 20:58:15.960067 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:15.959988    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt" event={"ID":"e6679e73-069b-424d-9cce-972deda45aa6","Type":"ContainerStarted","Data":"937ad6bf44b6827466d0462eb72cf77498193696844a842b46578cb266b6b24c"}
Sep 09 20:58:16.002611 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:16.002580    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4x2c5\""
Sep 09 20:58:16.064690 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:16.064665    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 20:58:16.069777 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:16.069757    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4751359f-6f18-4dca-ab21-69d8583c0bc3-cert\") pod \"kserve-controller-manager-d6f4b65b8-smxn7\" (UID: \"4751359f-6f18-4dca-ab21-69d8583c0bc3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:16.164069 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:16.164039    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:16.510095 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:16.510048    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"]
Sep 09 20:58:16.565074 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:58:16.565037    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4751359f_6f18_4dca_ab21_69d8583c0bc3.slice/crio-35038ca8430ee78ac8a0025a1f011437bc2297aa16e2f5b1b6e9e625251b3bdd WatchSource:0}: Error finding container 35038ca8430ee78ac8a0025a1f011437bc2297aa16e2f5b1b6e9e625251b3bdd: Status 404 returned error can't find the container with id 35038ca8430ee78ac8a0025a1f011437bc2297aa16e2f5b1b6e9e625251b3bdd
Sep 09 20:58:16.968385 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:16.968347    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7" event={"ID":"4751359f-6f18-4dca-ab21-69d8583c0bc3","Type":"ContainerStarted","Data":"35038ca8430ee78ac8a0025a1f011437bc2297aa16e2f5b1b6e9e625251b3bdd"}
Sep 09 20:58:17.919860 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:17.919827    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-dgmjp"
Sep 09 20:58:17.927191 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:17.927162    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"]
Sep 09 20:58:17.927389 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:17.927367    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" podUID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" containerName="discovery" containerID="cri-o://8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2" gracePeriod=30
Sep 09 20:58:18.856584 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.856562    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:18.927092 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.927074    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ssnpl"
Sep 09 20:58:18.978392 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.978365    2548 generic.go:356] "Generic (PLEG): container finished" podID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" containerID="8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2" exitCode=0
Sep 09 20:58:18.978518 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.978420    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"
Sep 09 20:58:18.978518 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.978453    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" event={"ID":"7c174c7e-ab77-4cc9-b0e2-5156e1eed257","Type":"ContainerDied","Data":"8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2"}
Sep 09 20:58:18.978518 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.978505    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg" event={"ID":"7c174c7e-ab77-4cc9-b0e2-5156e1eed257","Type":"ContainerDied","Data":"515e0fc5630a7738b10a0f4375724b6c9c1600f04d7eb798224267cc442f1351"}
Sep 09 20:58:18.978688 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.978532    2548 scope.go:117] "RemoveContainer" containerID="8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2"
Sep 09 20:58:18.980331 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.980309    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt" event={"ID":"e6679e73-069b-424d-9cce-972deda45aa6","Type":"ContainerStarted","Data":"1ef747f94cc5adcc80a0a87fee41c9ebbd3ddc773a270c06e61b698b7d21d016"}
Sep 09 20:58:18.980450 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.980368    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:18.982026 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.982007    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7" event={"ID":"4751359f-6f18-4dca-ab21-69d8583c0bc3","Type":"ContainerStarted","Data":"86f6ed247855026232abc3d7a21fe49f8dda01cd7fcb8d66a83a0d8f70675db0"}
Sep 09 20:58:18.982153 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.982138    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:58:18.987385 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.987369    2548 scope.go:117] "RemoveContainer" containerID="8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2"
Sep 09 20:58:18.987670 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:58:18.987641    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2\": container with ID starting with 8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2 not found: ID does not exist" containerID="8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2"
Sep 09 20:58:18.987726 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:18.987672    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2"} err="failed to get container status \"8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2\": rpc error: code = NotFound desc = could not find container \"8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2\": container with ID starting with 8ebaeddffb8241654336dea4eee78d3e5d347e453c0642a5d07a9f7c3480f7e2 not found: ID does not exist"
Sep 09 20:58:19.002020 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002001    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-ca-configmap\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002087 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002039    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-token\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002087 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002060    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-local-certs\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002201 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002090    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-cacerts\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002201 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002126    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-dns-cert\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002201 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002192    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-kubeconfig\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002321 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002217    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j46x6\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-kube-api-access-j46x6\") pod \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\" (UID: \"7c174c7e-ab77-4cc9-b0e2-5156e1eed257\") "
Sep 09 20:58:19.002470 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.002441    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:58:19.004747 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.004701    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-cacerts" (OuterVolumeSpecName: "cacerts") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:19.004838 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.004804    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:19.004838 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.004813    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-local-certs" (OuterVolumeSpecName: "local-certs") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:58:19.004923 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.004855    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:19.004923 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.004884    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-kube-api-access-j46x6" (OuterVolumeSpecName: "kube-api-access-j46x6") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "kube-api-access-j46x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:58:19.004923 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.004896    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7c174c7e-ab77-4cc9-b0e2-5156e1eed257" (UID: "7c174c7e-ab77-4cc9-b0e2-5156e1eed257"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:58:19.102988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.102916    2548 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-cacerts\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.102988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.102946    2548 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-dns-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.102988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.102959    2548 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-kubeconfig\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.102988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.102973    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46x6\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-kube-api-access-j46x6\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.102988 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.102988    2548 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-csr-ca-configmap\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.103222 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.103002    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.103222 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.103015    2548 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c174c7e-ab77-4cc9-b0e2-5156e1eed257-local-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:58:19.211751 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.211688    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7" podStartSLOduration=2.903095103 podStartE2EDuration="5.211673949s" totalImagesPullingTime="2.308578846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:19.202383789 +0000 UTC m=+587.705213790" watchObservedRunningTime="2026-09-09 20:58:19.211673949 +0000 UTC m=+587.714503949"
Sep 09 20:58:19.265722 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.265662    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt" podStartSLOduration=2.0937083 podStartE2EDuration="5.26564457s" totalImagesPullingTime="3.17193627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:19.263672406 +0000 UTC m=+587.766502410" watchObservedRunningTime="2026-09-09 20:58:19.26564457 +0000 UTC m=+587.768474585"
Sep 09 20:58:19.307598 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.307566    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"]
Sep 09 20:58:19.329320 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.329292    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-k76qg"]
Sep 09 20:58:19.967518 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:19.967485    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" path="/var/lib/kubelet/pods/7c174c7e-ab77-4cc9-b0e2-5156e1eed257/volumes"
Sep 09 20:58:31.947274 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:31.947242    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:58:31.947815 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:31.947606    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 20:58:49.988693 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:49.988660    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cd2bt"
Sep 09 20:58:50.000363 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:58:49.991789    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smxn7"
Sep 09 20:59:16.829442 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.829404    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"]
Sep 09 20:59:16.829957 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.829888    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" containerName="discovery"
Sep 09 20:59:16.829957 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.829909    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" containerName="discovery"
Sep 09 20:59:16.830069 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.829991    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c174c7e-ab77-4cc9-b0e2-5156e1eed257" containerName="discovery"
Sep 09 20:59:16.834424 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.834402    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:16.837790 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.837768    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:59:16.837878 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.837817    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:59:16.837878 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.837857    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 20:59:16.837878 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.837870    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dvpnd\""
Sep 09 20:59:16.847261 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.847241    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"]
Sep 09 20:59:16.967595 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:16.967564    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6d9l\" (UniqueName: \"kubernetes.io/projected/ffa34442-1587-4aa9-ac6c-1190b0a3e7fc-kube-api-access-j6d9l\") pod \"dns-operator-controller-manager-7b84cd7c9c-sblxn\" (UID: \"ffa34442-1587-4aa9-ac6c-1190b0a3e7fc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:17.068534 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:17.068504    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6d9l\" (UniqueName: \"kubernetes.io/projected/ffa34442-1587-4aa9-ac6c-1190b0a3e7fc-kube-api-access-j6d9l\") pod \"dns-operator-controller-manager-7b84cd7c9c-sblxn\" (UID: \"ffa34442-1587-4aa9-ac6c-1190b0a3e7fc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:17.082024 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:17.081965    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6d9l\" (UniqueName: \"kubernetes.io/projected/ffa34442-1587-4aa9-ac6c-1190b0a3e7fc-kube-api-access-j6d9l\") pod \"dns-operator-controller-manager-7b84cd7c9c-sblxn\" (UID: \"ffa34442-1587-4aa9-ac6c-1190b0a3e7fc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:17.144842 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:17.144816    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:17.447519 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:17.447482    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"]
Sep 09 20:59:17.451850 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:17.451823    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa34442_1587_4aa9_ac6c_1190b0a3e7fc.slice/crio-8f9ac0f23f5334c44a5208357206d71f1aaae3c2cb74825c66538316cb20c372 WatchSource:0}: Error finding container 8f9ac0f23f5334c44a5208357206d71f1aaae3c2cb74825c66538316cb20c372: Status 404 returned error can't find the container with id 8f9ac0f23f5334c44a5208357206d71f1aaae3c2cb74825c66538316cb20c372
Sep 09 20:59:17.454059 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:17.454041    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:59:18.221746 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:18.221698    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn" event={"ID":"ffa34442-1587-4aa9-ac6c-1190b0a3e7fc","Type":"ContainerStarted","Data":"8f9ac0f23f5334c44a5208357206d71f1aaae3c2cb74825c66538316cb20c372"}
Sep 09 20:59:20.231926 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:20.231887    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn" event={"ID":"ffa34442-1587-4aa9-ac6c-1190b0a3e7fc","Type":"ContainerStarted","Data":"a5f1a0f129619a70074d6ed7eb2dc0fcb34258d8dd493e048aa0e86769ed4c07"}
Sep 09 20:59:20.232260 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:20.231961    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:20.257933 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:20.257858    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn" podStartSLOduration=1.710998638 podStartE2EDuration="4.257843067s" totalImagesPullingTime="2.546844429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:20.255199432 +0000 UTC m=+648.758029433" watchObservedRunningTime="2026-09-09 20:59:20.257843067 +0000 UTC m=+648.760673068"
Sep 09 20:59:22.219905 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.219872    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"]
Sep 09 20:59:22.223188 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.223171    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.230480 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.230463    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ks4mt\""
Sep 09 20:59:22.252199 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.252177    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"]
Sep 09 20:59:22.310126 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.310100    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8tr\" (UniqueName: \"kubernetes.io/projected/6a51d48c-7708-40e6-afe1-ed37c1639bcb-kube-api-access-gt8tr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.310229 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.310214    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a51d48c-7708-40e6-afe1-ed37c1639bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.411144 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.411111    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8tr\" (UniqueName: \"kubernetes.io/projected/6a51d48c-7708-40e6-afe1-ed37c1639bcb-kube-api-access-gt8tr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.411257 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.411209    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a51d48c-7708-40e6-afe1-ed37c1639bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.411562 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.411539    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a51d48c-7708-40e6-afe1-ed37c1639bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.424193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.424174    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8tr\" (UniqueName: \"kubernetes.io/projected/6a51d48c-7708-40e6-afe1-ed37c1639bcb-kube-api-access-gt8tr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.533528 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.533474    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:22.983771 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:22.983727    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a51d48c_7708_40e6_afe1_ed37c1639bcb.slice/crio-588a18d1f742a5c992846ce508066728a49320e7ddc3d26366773f5ec742d4d0 WatchSource:0}: Error finding container 588a18d1f742a5c992846ce508066728a49320e7ddc3d26366773f5ec742d4d0: Status 404 returned error can't find the container with id 588a18d1f742a5c992846ce508066728a49320e7ddc3d26366773f5ec742d4d0
Sep 09 20:59:22.991758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:22.991720    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"]
Sep 09 20:59:23.245190 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:23.245116    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz" event={"ID":"6a51d48c-7708-40e6-afe1-ed37c1639bcb","Type":"ContainerStarted","Data":"588a18d1f742a5c992846ce508066728a49320e7ddc3d26366773f5ec742d4d0"}
Sep 09 20:59:25.387585 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.387554    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5mrdt"]
Sep 09 20:59:25.391477 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.391455    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:25.394268 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.394240    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xb8fl\""
Sep 09 20:59:25.409174 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.409148    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5mrdt"]
Sep 09 20:59:25.536346 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.536314    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl48t\" (UniqueName: \"kubernetes.io/projected/4fa60691-9cae-423a-a153-bed4e83bd0d4-kube-api-access-hl48t\") pod \"authorino-operator-76b478bd78-5mrdt\" (UID: \"4fa60691-9cae-423a-a153-bed4e83bd0d4\") " pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:25.637937 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.637848    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl48t\" (UniqueName: \"kubernetes.io/projected/4fa60691-9cae-423a-a153-bed4e83bd0d4-kube-api-access-hl48t\") pod \"authorino-operator-76b478bd78-5mrdt\" (UID: \"4fa60691-9cae-423a-a153-bed4e83bd0d4\") " pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:25.693871 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.693838    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl48t\" (UniqueName: \"kubernetes.io/projected/4fa60691-9cae-423a-a153-bed4e83bd0d4-kube-api-access-hl48t\") pod \"authorino-operator-76b478bd78-5mrdt\" (UID: \"4fa60691-9cae-423a-a153-bed4e83bd0d4\") " pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:25.703538 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:25.703514    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:26.903622 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:26.903598    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5mrdt"]
Sep 09 20:59:26.905255 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:26.905230    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa60691_9cae_423a_a153_bed4e83bd0d4.slice/crio-0ad9cce85646e1e61a70cf7220ab1fde0a7291d757694b8b612138b01b4ead75 WatchSource:0}: Error finding container 0ad9cce85646e1e61a70cf7220ab1fde0a7291d757694b8b612138b01b4ead75: Status 404 returned error can't find the container with id 0ad9cce85646e1e61a70cf7220ab1fde0a7291d757694b8b612138b01b4ead75
Sep 09 20:59:27.265978 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:27.265941    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz" event={"ID":"6a51d48c-7708-40e6-afe1-ed37c1639bcb","Type":"ContainerStarted","Data":"f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961"}
Sep 09 20:59:27.266159 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:27.266093    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:27.272979 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:27.272940    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt" event={"ID":"4fa60691-9cae-423a-a153-bed4e83bd0d4","Type":"ContainerStarted","Data":"0ad9cce85646e1e61a70cf7220ab1fde0a7291d757694b8b612138b01b4ead75"}
Sep 09 20:59:27.319816 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:27.319771    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz" podStartSLOduration=1.621541312 podStartE2EDuration="5.319759183s" totalImagesPullingTime="3.698217871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:27.316472687 +0000 UTC m=+655.819302687" watchObservedRunningTime="2026-09-09 20:59:27.319759183 +0000 UTC m=+655.822589184"
Sep 09 20:59:28.417252 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.417221    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"]
Sep 09 20:59:28.420610 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.420590    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:28.423878 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.423858    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-964dt\""
Sep 09 20:59:28.446204 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.446174    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"]
Sep 09 20:59:28.564799 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.564769    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvrw\" (UniqueName: \"kubernetes.io/projected/b1607e5b-c517-4aa6-98a3-e556cfa7a236-kube-api-access-hsvrw\") pod \"limitador-operator-controller-manager-74495c69f8-t29ml\" (UID: \"b1607e5b-c517-4aa6-98a3-e556cfa7a236\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:28.666179 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.666105    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvrw\" (UniqueName: \"kubernetes.io/projected/b1607e5b-c517-4aa6-98a3-e556cfa7a236-kube-api-access-hsvrw\") pod \"limitador-operator-controller-manager-74495c69f8-t29ml\" (UID: \"b1607e5b-c517-4aa6-98a3-e556cfa7a236\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:28.686135 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.686104    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvrw\" (UniqueName: \"kubernetes.io/projected/b1607e5b-c517-4aa6-98a3-e556cfa7a236-kube-api-access-hsvrw\") pod \"limitador-operator-controller-manager-74495c69f8-t29ml\" (UID: \"b1607e5b-c517-4aa6-98a3-e556cfa7a236\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:28.729870 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:28.729847    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:29.019052 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:29.019029    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"]
Sep 09 20:59:29.020960 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:29.020923    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1607e5b_c517_4aa6_98a3_e556cfa7a236.slice/crio-258b25d4e7a019105ca6a9958e84cb496ca3fa1ae237f0118a4ecdea89f6bf5c WatchSource:0}: Error finding container 258b25d4e7a019105ca6a9958e84cb496ca3fa1ae237f0118a4ecdea89f6bf5c: Status 404 returned error can't find the container with id 258b25d4e7a019105ca6a9958e84cb496ca3fa1ae237f0118a4ecdea89f6bf5c
Sep 09 20:59:29.282448 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:29.282356    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml" event={"ID":"b1607e5b-c517-4aa6-98a3-e556cfa7a236","Type":"ContainerStarted","Data":"258b25d4e7a019105ca6a9958e84cb496ca3fa1ae237f0118a4ecdea89f6bf5c"}
Sep 09 20:59:29.284052 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:29.284029    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt" event={"ID":"4fa60691-9cae-423a-a153-bed4e83bd0d4","Type":"ContainerStarted","Data":"ac50164771dc8c8a384c961c470121c070e83a3898cd2c4715d88043f7c53e05"}
Sep 09 20:59:29.284167 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:29.284087    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:29.313522 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:29.313481    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt" podStartSLOduration=2.8992077050000002 podStartE2EDuration="4.313465715s" totalImagesPullingTime="1.41425801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:29.31299447 +0000 UTC m=+657.815824472" watchObservedRunningTime="2026-09-09 20:59:29.313465715 +0000 UTC m=+657.816295715"
Sep 09 20:59:31.238097 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:31.238065    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sblxn"
Sep 09 20:59:31.297042 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:31.297002    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml" event={"ID":"b1607e5b-c517-4aa6-98a3-e556cfa7a236","Type":"ContainerStarted","Data":"7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95"}
Sep 09 20:59:31.297177 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:31.297144    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:31.322335 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:31.322291    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml" podStartSLOduration=1.8102745869999999 podStartE2EDuration="3.322276154s" totalImagesPullingTime="1.512001567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:31.320543689 +0000 UTC m=+659.823373690" watchObservedRunningTime="2026-09-09 20:59:31.322276154 +0000 UTC m=+659.825106154"
Sep 09 20:59:38.280263 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:38.280230    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:40.292790 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.292758    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-5mrdt"
Sep 09 20:59:40.812121 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.812085    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"]
Sep 09 20:59:40.812316 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.812278    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" containerName="manager" containerID="cri-o://f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961" gracePeriod=2
Sep 09 20:59:40.832456 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.832428    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"]
Sep 09 20:59:40.847856 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.847824    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"]
Sep 09 20:59:40.848101 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.848073    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" containerName="manager" containerID="cri-o://7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95" gracePeriod=2
Sep 09 20:59:40.850007 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.849985    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:40.850908 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.850879    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"]
Sep 09 20:59:40.851351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.851294    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" containerName="manager"
Sep 09 20:59:40.851351 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.851315    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" containerName="manager"
Sep 09 20:59:40.851496 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.851417    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" containerName="manager"
Sep 09 20:59:40.854875 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.854856    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:40.856676 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:40.856648    2548 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"6a51d48c-7708-40e6-afe1-ed37c1639bcb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\": pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:40.861615 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.861591    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"]
Sep 09 20:59:40.866664 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:40.866635    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:40.872811 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.872785    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"]
Sep 09 20:59:40.891090 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.891064    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"]
Sep 09 20:59:40.891619 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.891595    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" containerName="manager"
Sep 09 20:59:40.891619 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.891621    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" containerName="manager"
Sep 09 20:59:40.891778 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.891704    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" containerName="manager"
Sep 09 20:59:40.900458 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.900427    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:40.911630 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:40.911601    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:40.915003 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.914977    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"]
Sep 09 20:59:40.936977 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:40.936936    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:40.966642 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.966615    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rjq\" (UniqueName: \"kubernetes.io/projected/d0f4d726-a804-49ff-918f-5501f812586a-kube-api-access-g4rjq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5btww\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:40.966826 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:40.966721    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f4d726-a804-49ff-918f-5501f812586a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5btww\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:41.067502 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.067426    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rjq\" (UniqueName: \"kubernetes.io/projected/d0f4d726-a804-49ff-918f-5501f812586a-kube-api-access-g4rjq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5btww\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:41.067703 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.067531    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhnl\" (UniqueName: \"kubernetes.io/projected/8905ab88-e57b-45a3-9a35-e26b9e26b53a-kube-api-access-nnhnl\") pod \"limitador-operator-controller-manager-74495c69f8-26vxx\" (UID: \"8905ab88-e57b-45a3-9a35-e26b9e26b53a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:41.067703 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.067583    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f4d726-a804-49ff-918f-5501f812586a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5btww\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:41.068052 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.068028    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f4d726-a804-49ff-918f-5501f812586a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5btww\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:41.081529 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.081503    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rjq\" (UniqueName: \"kubernetes.io/projected/d0f4d726-a804-49ff-918f-5501f812586a-kube-api-access-g4rjq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-5btww\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:41.097705 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.097680    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.099793 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.099776    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.102780 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.102758    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.104436 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.104414    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.106774 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.106756    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.108236 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.108217    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.168542 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.168515    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhnl\" (UniqueName: \"kubernetes.io/projected/8905ab88-e57b-45a3-9a35-e26b9e26b53a-kube-api-access-nnhnl\") pod \"limitador-operator-controller-manager-74495c69f8-26vxx\" (UID: \"8905ab88-e57b-45a3-9a35-e26b9e26b53a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:41.178772 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.178750    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhnl\" (UniqueName: \"kubernetes.io/projected/8905ab88-e57b-45a3-9a35-e26b9e26b53a-kube-api-access-nnhnl\") pod \"limitador-operator-controller-manager-74495c69f8-26vxx\" (UID: \"8905ab88-e57b-45a3-9a35-e26b9e26b53a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:41.266312 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.266267    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:41.269213 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.269190    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt8tr\" (UniqueName: \"kubernetes.io/projected/6a51d48c-7708-40e6-afe1-ed37c1639bcb-kube-api-access-gt8tr\") pod \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") "
Sep 09 20:59:41.269343 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.269248    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a51d48c-7708-40e6-afe1-ed37c1639bcb-extensions-socket-volume\") pod \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\" (UID: \"6a51d48c-7708-40e6-afe1-ed37c1639bcb\") "
Sep 09 20:59:41.269343 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.269290    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsvrw\" (UniqueName: \"kubernetes.io/projected/b1607e5b-c517-4aa6-98a3-e556cfa7a236-kube-api-access-hsvrw\") pod \"b1607e5b-c517-4aa6-98a3-e556cfa7a236\" (UID: \"b1607e5b-c517-4aa6-98a3-e556cfa7a236\") "
Sep 09 20:59:41.269686 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.269654    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a51d48c-7708-40e6-afe1-ed37c1639bcb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a51d48c-7708-40e6-afe1-ed37c1639bcb" (UID: "6a51d48c-7708-40e6-afe1-ed37c1639bcb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:41.271471 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.271437    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1607e5b-c517-4aa6-98a3-e556cfa7a236-kube-api-access-hsvrw" (OuterVolumeSpecName: "kube-api-access-hsvrw") pod "b1607e5b-c517-4aa6-98a3-e556cfa7a236" (UID: "b1607e5b-c517-4aa6-98a3-e556cfa7a236"). InnerVolumeSpecName "kube-api-access-hsvrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:41.271552 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.271480    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a51d48c-7708-40e6-afe1-ed37c1639bcb-kube-api-access-gt8tr" (OuterVolumeSpecName: "kube-api-access-gt8tr") pod "6a51d48c-7708-40e6-afe1-ed37c1639bcb" (UID: "6a51d48c-7708-40e6-afe1-ed37c1639bcb"). InnerVolumeSpecName "kube-api-access-gt8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:41.275389 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.272969    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:41.344311 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.344277    2548 generic.go:356] "Generic (PLEG): container finished" podID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" containerID="f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961" exitCode=0
Sep 09 20:59:41.344749 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.344323    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.344749 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.344362    2548 scope.go:117] "RemoveContainer" containerID="f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961"
Sep 09 20:59:41.345674 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.345658    2548 generic.go:356] "Generic (PLEG): container finished" podID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" containerID="7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95" exitCode=0
Sep 09 20:59:41.345777 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.345714    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.347819 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.347560    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.349766 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.349560    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.352616 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.352593    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.354527 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.354492    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.355587 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.355565    2548 scope.go:117] "RemoveContainer" containerID="f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961"
Sep 09 20:59:41.355882 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.355864    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961\": container with ID starting with f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961 not found: ID does not exist" containerID="f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961"
Sep 09 20:59:41.355972 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.355893    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961"} err="failed to get container status \"f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961\": rpc error: code = NotFound desc = could not find container \"f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961\": container with ID starting with f2bffe9f0c4f54b28c59c95a0137dd7777bddf1f9e9d9895b0185d00348a9961 not found: ID does not exist"
Sep 09 20:59:41.355972 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.355923    2548 scope.go:117] "RemoveContainer" containerID="7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95"
Sep 09 20:59:41.357417 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.357392    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.359070 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.359045    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.361016 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.360991    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.364089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.364068    2548 scope.go:117] "RemoveContainer" containerID="7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95"
Sep 09 20:59:41.364370 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.364350    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95\": container with ID starting with 7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95 not found: ID does not exist" containerID="7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95"
Sep 09 20:59:41.364427 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.364380    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95"} err="failed to get container status \"7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95\": rpc error: code = NotFound desc = could not find container \"7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95\": container with ID starting with 7ddffa43037b667eca6eae2ca00390f25a9a169e999bf3d75f169ef2abaffd95 not found: ID does not exist"
Sep 09 20:59:41.369297 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.369273    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.370756 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.370709    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8tr\" (UniqueName: \"kubernetes.io/projected/6a51d48c-7708-40e6-afe1-ed37c1639bcb-kube-api-access-gt8tr\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:59:41.370857 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.370758    2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a51d48c-7708-40e6-afe1-ed37c1639bcb-extensions-socket-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:59:41.370857 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.370768    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsvrw\" (UniqueName: \"kubernetes.io/projected/b1607e5b-c517-4aa6-98a3-e556cfa7a236-kube-api-access-hsvrw\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:59:41.562328 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.562228    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"]
Sep 09 20:59:41.564033 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.564011    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"]
Sep 09 20:59:41.565625 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:41.565595    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8905ab88_e57b_45a3_9a35_e26b9e26b53a.slice/crio-4afb032fbe0de820ee1e7fce9b668ae3fbe0b204c2d429b6f778af35426e5722 WatchSource:0}: Error finding container 4afb032fbe0de820ee1e7fce9b668ae3fbe0b204c2d429b6f778af35426e5722: Status 404 returned error can't find the container with id 4afb032fbe0de820ee1e7fce9b668ae3fbe0b204c2d429b6f778af35426e5722
Sep 09 20:59:41.566135 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:41.566113    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f4d726_a804_49ff_918f_5501f812586a.slice/crio-423700999c13029582ec793f7b102fb726c8186a2cd079a711fb6de5ed06c637 WatchSource:0}: Error finding container 423700999c13029582ec793f7b102fb726c8186a2cd079a711fb6de5ed06c637: Status 404 returned error can't find the container with id 423700999c13029582ec793f7b102fb726c8186a2cd079a711fb6de5ed06c637
Sep 09 20:59:41.967752 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.967713    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" path="/var/lib/kubelet/pods/6a51d48c-7708-40e6-afe1-ed37c1639bcb/volumes"
Sep 09 20:59:41.968119 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.968099    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" path="/var/lib/kubelet/pods/b1607e5b-c517-4aa6-98a3-e556cfa7a236/volumes"
Sep 09 20:59:41.969125 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.969095    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-z4xkz\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="6a51d48c-7708-40e6-afe1-ed37c1639bcb" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-z4xkz"
Sep 09 20:59:41.971858 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:41.971830    2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t29ml\" is forbidden: User \"system:node:ip-10-0-134-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-62.ec2.internal' and this object" podUID="b1607e5b-c517-4aa6-98a3-e556cfa7a236" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t29ml"
Sep 09 20:59:41.997207 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:41.997183    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"]
Sep 09 20:59:42.000800 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.000783    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.040039 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.040014    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"]
Sep 09 20:59:42.178154 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.178070    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j56qk\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.178283 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.178171    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4wr\" (UniqueName: \"kubernetes.io/projected/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-kube-api-access-xk4wr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j56qk\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.279034 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.279000    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j56qk\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.279193 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.279144    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4wr\" (UniqueName: \"kubernetes.io/projected/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-kube-api-access-xk4wr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j56qk\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.279381 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.279361    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j56qk\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.298399 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.298373    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4wr\" (UniqueName: \"kubernetes.io/projected/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-kube-api-access-xk4wr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-j56qk\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.309980 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.309956    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:42.353758 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.353701    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx" event={"ID":"8905ab88-e57b-45a3-9a35-e26b9e26b53a","Type":"ContainerStarted","Data":"817b16525b0d8c37c714399232dbeda83f9586f0663d8bc9d47b66f66af53294"}
Sep 09 20:59:42.354231 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.353794    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:42.354231 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.353821    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx" event={"ID":"8905ab88-e57b-45a3-9a35-e26b9e26b53a","Type":"ContainerStarted","Data":"4afb032fbe0de820ee1e7fce9b668ae3fbe0b204c2d429b6f778af35426e5722"}
Sep 09 20:59:42.358059 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.358033    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww" event={"ID":"d0f4d726-a804-49ff-918f-5501f812586a","Type":"ContainerStarted","Data":"3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe"}
Sep 09 20:59:42.358189 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.358085    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww" event={"ID":"d0f4d726-a804-49ff-918f-5501f812586a","Type":"ContainerStarted","Data":"423700999c13029582ec793f7b102fb726c8186a2cd079a711fb6de5ed06c637"}
Sep 09 20:59:42.358189 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.358128    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:42.376606 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.376550    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx" podStartSLOduration=2.376528346 podStartE2EDuration="2.376528346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:42.375480006 +0000 UTC m=+670.878310007" watchObservedRunningTime="2026-09-09 20:59:42.376528346 +0000 UTC m=+670.879358348"
Sep 09 20:59:42.399267 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.399223    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww" podStartSLOduration=2.399208007 podStartE2EDuration="2.399208007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:42.398149925 +0000 UTC m=+670.900979925" watchObservedRunningTime="2026-09-09 20:59:42.399208007 +0000 UTC m=+670.902038007"
Sep 09 20:59:42.585780 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:42.585724    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"]
Sep 09 20:59:42.589457 ip-10-0-134-62 kubenswrapper[2548]: W0909 20:59:42.589427    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc2b8c5_5c34_4c0e_846d_0d3e82688217.slice/crio-826622e4238b84e5f4a6727af20978ddaea9b2b47ebf6820b6d7cd2495c8336a WatchSource:0}: Error finding container 826622e4238b84e5f4a6727af20978ddaea9b2b47ebf6820b6d7cd2495c8336a: Status 404 returned error can't find the container with id 826622e4238b84e5f4a6727af20978ddaea9b2b47ebf6820b6d7cd2495c8336a
Sep 09 20:59:43.365277 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:43.365230    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk" event={"ID":"5dc2b8c5-5c34-4c0e-846d-0d3e82688217","Type":"ContainerStarted","Data":"a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2"}
Sep 09 20:59:43.365795 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:43.365300    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk" event={"ID":"5dc2b8c5-5c34-4c0e-846d-0d3e82688217","Type":"ContainerStarted","Data":"826622e4238b84e5f4a6727af20978ddaea9b2b47ebf6820b6d7cd2495c8336a"}
Sep 09 20:59:43.365795 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:43.365412    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:43.386054 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:43.386005    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk" podStartSLOduration=2.385989238 podStartE2EDuration="2.385989238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:43.385056053 +0000 UTC m=+671.887886054" watchObservedRunningTime="2026-09-09 20:59:43.385989238 +0000 UTC m=+671.888819239"
Sep 09 20:59:53.367775 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:53.367719    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-26vxx"
Sep 09 20:59:53.368190 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:53.367837    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:54.371886 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.371851    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 20:59:54.472362 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.472317    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"]
Sep 09 20:59:54.472551 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.472513    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww" podUID="d0f4d726-a804-49ff-918f-5501f812586a" containerName="manager" containerID="cri-o://3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe" gracePeriod=10
Sep 09 20:59:54.722482 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.722459    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:54.785274 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.785068    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f4d726-a804-49ff-918f-5501f812586a-extensions-socket-volume\") pod \"d0f4d726-a804-49ff-918f-5501f812586a\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") "
Sep 09 20:59:54.785274 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.785114    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4rjq\" (UniqueName: \"kubernetes.io/projected/d0f4d726-a804-49ff-918f-5501f812586a-kube-api-access-g4rjq\") pod \"d0f4d726-a804-49ff-918f-5501f812586a\" (UID: \"d0f4d726-a804-49ff-918f-5501f812586a\") "
Sep 09 20:59:54.785486 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.785464    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f4d726-a804-49ff-918f-5501f812586a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d0f4d726-a804-49ff-918f-5501f812586a" (UID: "d0f4d726-a804-49ff-918f-5501f812586a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:59:54.787190 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.787158    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f4d726-a804-49ff-918f-5501f812586a-kube-api-access-g4rjq" (OuterVolumeSpecName: "kube-api-access-g4rjq") pod "d0f4d726-a804-49ff-918f-5501f812586a" (UID: "d0f4d726-a804-49ff-918f-5501f812586a"). InnerVolumeSpecName "kube-api-access-g4rjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:54.885895 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.885824    2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f4d726-a804-49ff-918f-5501f812586a-extensions-socket-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:59:54.885895 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:54.885849    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rjq\" (UniqueName: \"kubernetes.io/projected/d0f4d726-a804-49ff-918f-5501f812586a-kube-api-access-g4rjq\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 20:59:55.421089 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.421052    2548 generic.go:356] "Generic (PLEG): container finished" podID="d0f4d726-a804-49ff-918f-5501f812586a" containerID="3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe" exitCode=0
Sep 09 20:59:55.421469 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.421100    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"
Sep 09 20:59:55.421469 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.421137    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww" event={"ID":"d0f4d726-a804-49ff-918f-5501f812586a","Type":"ContainerDied","Data":"3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe"}
Sep 09 20:59:55.421469 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.421196    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww" event={"ID":"d0f4d726-a804-49ff-918f-5501f812586a","Type":"ContainerDied","Data":"423700999c13029582ec793f7b102fb726c8186a2cd079a711fb6de5ed06c637"}
Sep 09 20:59:55.421469 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.421217    2548 scope.go:117] "RemoveContainer" containerID="3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe"
Sep 09 20:59:55.429402 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.429382    2548 scope.go:117] "RemoveContainer" containerID="3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe"
Sep 09 20:59:55.429664 ip-10-0-134-62 kubenswrapper[2548]: E0909 20:59:55.429643    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe\": container with ID starting with 3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe not found: ID does not exist" containerID="3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe"
Sep 09 20:59:55.429716 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.429671    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe"} err="failed to get container status \"3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe\": rpc error: code = NotFound desc = could not find container \"3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe\": container with ID starting with 3415ca1ae6895a4d6001f3aa49d489c3321d205ed89f8f7f0814e9701832eebe not found: ID does not exist"
Sep 09 20:59:55.443434 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.443410    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"]
Sep 09 20:59:55.447181 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.447158    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-5btww"]
Sep 09 20:59:55.968262 ip-10-0-134-62 kubenswrapper[2548]: I0909 20:59:55.968231    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0f4d726-a804-49ff-918f-5501f812586a" path="/var/lib/kubelet/pods/d0f4d726-a804-49ff-918f-5501f812586a/volumes"
Sep 09 21:00:11.229693 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.229660    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"]
Sep 09 21:00:11.230071 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.230043    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0f4d726-a804-49ff-918f-5501f812586a" containerName="manager"
Sep 09 21:00:11.230071 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.230055    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0f4d726-a804-49ff-918f-5501f812586a" containerName="manager"
Sep 09 21:00:11.230187 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.230107    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0f4d726-a804-49ff-918f-5501f812586a" containerName="manager"
Sep 09 21:00:11.233101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.233080    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.235322 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.235300    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d5kqw\""
Sep 09 21:00:11.301125 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.301092    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"]
Sep 09 21:00:11.325323 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325289    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325340    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325358    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325411    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325476    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325500    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325572    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325618    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.325840 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.325667    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft429\" (UniqueName: \"kubernetes.io/projected/ffd75c7e-ad26-454b-a660-e8e41929a0d5-kube-api-access-ft429\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426451 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426416    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426451 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426450    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426469    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426592    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426634    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426679    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426925 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426702    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426925 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426755    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft429\" (UniqueName: \"kubernetes.io/projected/ffd75c7e-ad26-454b-a660-e8e41929a0d5-kube-api-access-ft429\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.426925 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426891    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.427077 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.426922    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.427166 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.427146    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.427217 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.427160    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.427217 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.427188    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.427478 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.427458    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.429107 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.429084    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.429199 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.429092    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.456894 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.456869    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd75c7e-ad26-454b-a660-e8e41929a0d5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.457020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.457001    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft429\" (UniqueName: \"kubernetes.io/projected/ffd75c7e-ad26-454b-a660-e8e41929a0d5-kube-api-access-ft429\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nrds4\" (UID: \"ffd75c7e-ad26-454b-a660-e8e41929a0d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.543404 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.543331    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:11.821123 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.821100    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"]
Sep 09 21:00:11.823810 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:00:11.823783    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd75c7e_ad26_454b_a660_e8e41929a0d5.slice/crio-8a74fb8f767d4bc3a81c2e1edcb66216c8ba1844d9a9060eab300152720b874b WatchSource:0}: Error finding container 8a74fb8f767d4bc3a81c2e1edcb66216c8ba1844d9a9060eab300152720b874b: Status 404 returned error can't find the container with id 8a74fb8f767d4bc3a81c2e1edcb66216c8ba1844d9a9060eab300152720b874b
Sep 09 21:00:11.825705 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.825673    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:11.825807 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.825753    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:11.825807 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:11.825783    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:12.502577 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:12.502535    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" event={"ID":"ffd75c7e-ad26-454b-a660-e8e41929a0d5","Type":"ContainerStarted","Data":"29625935dccbf33620628fffd396df9f75789644e66de36ded1ffa16f3ad7589"}
Sep 09 21:00:12.502971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:12.502589    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" event={"ID":"ffd75c7e-ad26-454b-a660-e8e41929a0d5","Type":"ContainerStarted","Data":"8a74fb8f767d4bc3a81c2e1edcb66216c8ba1844d9a9060eab300152720b874b"}
Sep 09 21:00:12.529377 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:12.529326    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" podStartSLOduration=1.529297996 podStartE2EDuration="1.529297996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:12.527801562 +0000 UTC m=+701.030631567" watchObservedRunningTime="2026-09-09 21:00:12.529297996 +0000 UTC m=+701.032127998"
Sep 09 21:00:12.544221 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:12.544192    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:12.548644 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:12.548620    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:13.507505 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:13.507479    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:13.508511 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:13.508485    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:00:14.878000 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:14.877966    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cppjf"]
Sep 09 21:00:14.881539 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:14.881518    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:14.884134 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:14.884114    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8kvtd\""
Sep 09 21:00:14.896415 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:14.896391    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cppjf"]
Sep 09 21:00:14.956381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:14.956339    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhjg\" (UniqueName: \"kubernetes.io/projected/35093177-9e61-4324-afc6-cb82f29052b7-kube-api-access-dnhjg\") pod \"authorino-59c974bd48-cppjf\" (UID: \"35093177-9e61-4324-afc6-cb82f29052b7\") " pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:15.057655 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:15.057610    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhjg\" (UniqueName: \"kubernetes.io/projected/35093177-9e61-4324-afc6-cb82f29052b7-kube-api-access-dnhjg\") pod \"authorino-59c974bd48-cppjf\" (UID: \"35093177-9e61-4324-afc6-cb82f29052b7\") " pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:15.068244 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:15.068214    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhjg\" (UniqueName: \"kubernetes.io/projected/35093177-9e61-4324-afc6-cb82f29052b7-kube-api-access-dnhjg\") pod \"authorino-59c974bd48-cppjf\" (UID: \"35093177-9e61-4324-afc6-cb82f29052b7\") " pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:15.190825 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:15.190799    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:15.459981 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:15.459929    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cppjf"]
Sep 09 21:00:15.462250 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:00:15.462213    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35093177_9e61_4324_afc6_cb82f29052b7.slice/crio-145ebf18881682f858fcc2394ed08cd6490f415c10c381acb7763bbb7592d307 WatchSource:0}: Error finding container 145ebf18881682f858fcc2394ed08cd6490f415c10c381acb7763bbb7592d307: Status 404 returned error can't find the container with id 145ebf18881682f858fcc2394ed08cd6490f415c10c381acb7763bbb7592d307
Sep 09 21:00:15.518225 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:15.518179    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cppjf" event={"ID":"35093177-9e61-4324-afc6-cb82f29052b7","Type":"ContainerStarted","Data":"145ebf18881682f858fcc2394ed08cd6490f415c10c381acb7763bbb7592d307"}
Sep 09 21:00:17.532502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:17.532402    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cppjf" event={"ID":"35093177-9e61-4324-afc6-cb82f29052b7","Type":"ContainerStarted","Data":"8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f"}
Sep 09 21:00:17.551508 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:17.551463    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cppjf" podStartSLOduration=1.825175675 podStartE2EDuration="3.551448807s" totalImagesPullingTime="1.726273132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:17.550452503 +0000 UTC m=+706.053282504" watchObservedRunningTime="2026-09-09 21:00:17.551448807 +0000 UTC m=+706.054278807"
Sep 09 21:00:50.758368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.758332    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dclhd"]
Sep 09 21:00:50.763133 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.763108    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:50.765503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.765482    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 21:00:50.771453 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.771431    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dclhd"]
Sep 09 21:00:50.844986 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.844952    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9de22ff2-1884-4d8a-88b2-dffcf144b641-tls-cert\") pod \"authorino-6d4dfc649f-dclhd\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") " pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:50.845129 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.845040    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnlz\" (UniqueName: \"kubernetes.io/projected/9de22ff2-1884-4d8a-88b2-dffcf144b641-kube-api-access-ptnlz\") pod \"authorino-6d4dfc649f-dclhd\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") " pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:50.859461 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.859431    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dclhd"]
Sep 09 21:00:50.859669 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:00:50.859651    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ptnlz tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-dclhd" podUID="9de22ff2-1884-4d8a-88b2-dffcf144b641"
Sep 09 21:00:50.945998 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.945952    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnlz\" (UniqueName: \"kubernetes.io/projected/9de22ff2-1884-4d8a-88b2-dffcf144b641-kube-api-access-ptnlz\") pod \"authorino-6d4dfc649f-dclhd\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") " pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:50.946165 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.946015    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9de22ff2-1884-4d8a-88b2-dffcf144b641-tls-cert\") pod \"authorino-6d4dfc649f-dclhd\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") " pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:50.948535 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.948510    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9de22ff2-1884-4d8a-88b2-dffcf144b641-tls-cert\") pod \"authorino-6d4dfc649f-dclhd\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") " pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:50.954400 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:50.954373    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnlz\" (UniqueName: \"kubernetes.io/projected/9de22ff2-1884-4d8a-88b2-dffcf144b641-kube-api-access-ptnlz\") pod \"authorino-6d4dfc649f-dclhd\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") " pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:51.691663 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.691627    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:51.699609 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.699586    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:51.751956 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.751923    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9de22ff2-1884-4d8a-88b2-dffcf144b641-tls-cert\") pod \"9de22ff2-1884-4d8a-88b2-dffcf144b641\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") "
Sep 09 21:00:51.751956 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.751962    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptnlz\" (UniqueName: \"kubernetes.io/projected/9de22ff2-1884-4d8a-88b2-dffcf144b641-kube-api-access-ptnlz\") pod \"9de22ff2-1884-4d8a-88b2-dffcf144b641\" (UID: \"9de22ff2-1884-4d8a-88b2-dffcf144b641\") "
Sep 09 21:00:51.754174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.754144    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de22ff2-1884-4d8a-88b2-dffcf144b641-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9de22ff2-1884-4d8a-88b2-dffcf144b641" (UID: "9de22ff2-1884-4d8a-88b2-dffcf144b641"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:51.754174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.754157    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de22ff2-1884-4d8a-88b2-dffcf144b641-kube-api-access-ptnlz" (OuterVolumeSpecName: "kube-api-access-ptnlz") pod "9de22ff2-1884-4d8a-88b2-dffcf144b641" (UID: "9de22ff2-1884-4d8a-88b2-dffcf144b641"). InnerVolumeSpecName "kube-api-access-ptnlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:51.853078 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.853042    2548 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9de22ff2-1884-4d8a-88b2-dffcf144b641-tls-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:00:51.853078 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:51.853072    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnlz\" (UniqueName: \"kubernetes.io/projected/9de22ff2-1884-4d8a-88b2-dffcf144b641-kube-api-access-ptnlz\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:00:52.695720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:52.695682    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dclhd"
Sep 09 21:00:52.720141 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:52.720109    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dclhd"]
Sep 09 21:00:52.725461 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:52.725432    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dclhd"]
Sep 09 21:00:53.122093 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.122020    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cppjf"]
Sep 09 21:00:53.122500 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.122221    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cppjf" podUID="35093177-9e61-4324-afc6-cb82f29052b7" containerName="authorino" containerID="cri-o://8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f" gracePeriod=30
Sep 09 21:00:53.365475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.365450    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:53.463899 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.463876    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnhjg\" (UniqueName: \"kubernetes.io/projected/35093177-9e61-4324-afc6-cb82f29052b7-kube-api-access-dnhjg\") pod \"35093177-9e61-4324-afc6-cb82f29052b7\" (UID: \"35093177-9e61-4324-afc6-cb82f29052b7\") "
Sep 09 21:00:53.465868 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.465845    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35093177-9e61-4324-afc6-cb82f29052b7-kube-api-access-dnhjg" (OuterVolumeSpecName: "kube-api-access-dnhjg") pod "35093177-9e61-4324-afc6-cb82f29052b7" (UID: "35093177-9e61-4324-afc6-cb82f29052b7"). InnerVolumeSpecName "kube-api-access-dnhjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:53.564474 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.564446    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhjg\" (UniqueName: \"kubernetes.io/projected/35093177-9e61-4324-afc6-cb82f29052b7-kube-api-access-dnhjg\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:00:53.701385 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.701348    2548 generic.go:356] "Generic (PLEG): container finished" podID="35093177-9e61-4324-afc6-cb82f29052b7" containerID="8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f" exitCode=0
Sep 09 21:00:53.701534 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.701389    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cppjf"
Sep 09 21:00:53.701534 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.701435    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cppjf" event={"ID":"35093177-9e61-4324-afc6-cb82f29052b7","Type":"ContainerDied","Data":"8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f"}
Sep 09 21:00:53.701534 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.701473    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cppjf" event={"ID":"35093177-9e61-4324-afc6-cb82f29052b7","Type":"ContainerDied","Data":"145ebf18881682f858fcc2394ed08cd6490f415c10c381acb7763bbb7592d307"}
Sep 09 21:00:53.701534 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.701490    2548 scope.go:117] "RemoveContainer" containerID="8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f"
Sep 09 21:00:53.710499 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.710481    2548 scope.go:117] "RemoveContainer" containerID="8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f"
Sep 09 21:00:53.710775 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:00:53.710727    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f\": container with ID starting with 8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f not found: ID does not exist" containerID="8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f"
Sep 09 21:00:53.710823 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.710786    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f"} err="failed to get container status \"8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f\": rpc error: code = NotFound desc = could not find container \"8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f\": container with ID starting with 8f1e8edfcf0a79619aa5d0977478083ff5cd5c7bda7c5054071444b39ff2ed3f not found: ID does not exist"
Sep 09 21:00:53.725850 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.725793    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cppjf"]
Sep 09 21:00:53.735013 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.734987    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cppjf"]
Sep 09 21:00:53.967392 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.967359    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35093177-9e61-4324-afc6-cb82f29052b7" path="/var/lib/kubelet/pods/35093177-9e61-4324-afc6-cb82f29052b7/volumes"
Sep 09 21:00:53.967725 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:53.967712    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9de22ff2-1884-4d8a-88b2-dffcf144b641" path="/var/lib/kubelet/pods/9de22ff2-1884-4d8a-88b2-dffcf144b641/volumes"
Sep 09 21:00:57.958065 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.958025    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-868c87cfb8-7rd89"]
Sep 09 21:00:57.958595 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.958572    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35093177-9e61-4324-afc6-cb82f29052b7" containerName="authorino"
Sep 09 21:00:57.958677 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.958599    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="35093177-9e61-4324-afc6-cb82f29052b7" containerName="authorino"
Sep 09 21:00:57.958727 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.958692    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="35093177-9e61-4324-afc6-cb82f29052b7" containerName="authorino"
Sep 09 21:00:57.961657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.961634    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:57.965298 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.965272    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:00:57.965423 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.965317    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-gateway-identity\""
Sep 09 21:00:57.965423 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.965331    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f842q\""
Sep 09 21:00:57.965423 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.965412    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:00:57.965983 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.965966    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:00:57.976149 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:57.976120    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-868c87cfb8-7rd89"]
Sep 09 21:00:58.000284 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.000257    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.000421 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.000371    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.000421 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.000399    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqdm5\" (UniqueName: \"kubernetes.io/projected/d38d8b43-901d-46c6-86a7-2034e6d1e989-kube-api-access-sqdm5\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.101447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.101414    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.101447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.101459    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqdm5\" (UniqueName: \"kubernetes.io/projected/d38d8b43-901d-46c6-86a7-2034e6d1e989-kube-api-access-sqdm5\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.101715 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.101505    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.101715 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:00:58.101571    2548 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 09 21:00:58.101715 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:00:58.101627    2548 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 09 21:00:58.101715 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:00:58.101648    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs podName:d38d8b43-901d-46c6-86a7-2034e6d1e989 nodeName:}" failed. No retries permitted until 2026-09-09 21:00:58.601630374 +0000 UTC m=+747.104460356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs") pod "maas-controller-868c87cfb8-7rd89" (UID: "d38d8b43-901d-46c6-86a7-2034e6d1e989") : secret "maas-controller-metrics-tls" not found
Sep 09 21:00:58.101715 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:00:58.101677    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert podName:d38d8b43-901d-46c6-86a7-2034e6d1e989 nodeName:}" failed. No retries permitted until 2026-09-09 21:00:58.601661613 +0000 UTC m=+747.104491592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert") pod "maas-controller-868c87cfb8-7rd89" (UID: "d38d8b43-901d-46c6-86a7-2034e6d1e989") : secret "maas-controller-webhook-cert" not found
Sep 09 21:00:58.117531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.117501    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqdm5\" (UniqueName: \"kubernetes.io/projected/d38d8b43-901d-46c6-86a7-2034e6d1e989-kube-api-access-sqdm5\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.123298 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.123264    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f44688b45-2nrvb"]
Sep 09 21:00:58.126961 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.126941    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.144799 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.144774    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44688b45-2nrvb"]
Sep 09 21:00:58.202069 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.202037    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxpz\" (UniqueName: \"kubernetes.io/projected/0119f136-b1b4-4306-879f-825cdf185435-kube-api-access-sdxpz\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.202069 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.202069    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-metrics-certs\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.202236 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.202087    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-webhook-cert\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.302703 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.302628    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxpz\" (UniqueName: \"kubernetes.io/projected/0119f136-b1b4-4306-879f-825cdf185435-kube-api-access-sdxpz\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.302703 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.302661    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-metrics-certs\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.302703 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.302681    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-webhook-cert\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.305162 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.305131    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-webhook-cert\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.305162 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.305159    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-metrics-certs\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.314931 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.314905    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxpz\" (UniqueName: \"kubernetes.io/projected/0119f136-b1b4-4306-879f-825cdf185435-kube-api-access-sdxpz\") pod \"maas-controller-f44688b45-2nrvb\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") " pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.436769 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.436718    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:00:58.605399 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.605314    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.605399 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.605365    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.607708 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.607683    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.607843 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.607687    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert\") pod \"maas-controller-868c87cfb8-7rd89\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") " pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:58.712346 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.712267    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44688b45-2nrvb"]
Sep 09 21:00:58.714662 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:00:58.714636    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0119f136_b1b4_4306_879f_825cdf185435.slice/crio-4890345f42187b990ef3ced1c9387c0dbf5a2f365a2af3aa9dcd29d5826fed49 WatchSource:0}: Error finding container 4890345f42187b990ef3ced1c9387c0dbf5a2f365a2af3aa9dcd29d5826fed49: Status 404 returned error can't find the container with id 4890345f42187b990ef3ced1c9387c0dbf5a2f365a2af3aa9dcd29d5826fed49
Sep 09 21:00:58.726433 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.726408    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-2nrvb" event={"ID":"0119f136-b1b4-4306-879f-825cdf185435","Type":"ContainerStarted","Data":"4890345f42187b990ef3ced1c9387c0dbf5a2f365a2af3aa9dcd29d5826fed49"}
Sep 09 21:00:58.872955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:58.872866    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:00:59.146155 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:59.146042    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-868c87cfb8-7rd89"]
Sep 09 21:00:59.148671 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:00:59.148640    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38d8b43_901d_46c6_86a7_2034e6d1e989.slice/crio-d7994fe18955989f4b7b6981e74ffdeb3b5e685c27ffabe220e678d7af4b2992 WatchSource:0}: Error finding container d7994fe18955989f4b7b6981e74ffdeb3b5e685c27ffabe220e678d7af4b2992: Status 404 returned error can't find the container with id d7994fe18955989f4b7b6981e74ffdeb3b5e685c27ffabe220e678d7af4b2992
Sep 09 21:00:59.743357 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:00:59.743315    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868c87cfb8-7rd89" event={"ID":"d38d8b43-901d-46c6-86a7-2034e6d1e989","Type":"ContainerStarted","Data":"d7994fe18955989f4b7b6981e74ffdeb3b5e685c27ffabe220e678d7af4b2992"}
Sep 09 21:01:02.766187 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:02.766125    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868c87cfb8-7rd89" event={"ID":"d38d8b43-901d-46c6-86a7-2034e6d1e989","Type":"ContainerStarted","Data":"600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e"}
Sep 09 21:01:02.766673 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:02.766213    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:01:02.767824 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:02.767793    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-2nrvb" event={"ID":"0119f136-b1b4-4306-879f-825cdf185435","Type":"ContainerStarted","Data":"0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d"}
Sep 09 21:01:02.767946 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:02.767879    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:01:02.784755 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:02.784681    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-868c87cfb8-7rd89" podStartSLOduration=3.15396168 podStartE2EDuration="5.784665319s" totalImagesPullingTime="2.630703639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:02.782863583 +0000 UTC m=+751.285693600" watchObservedRunningTime="2026-09-09 21:01:02.784665319 +0000 UTC m=+751.287495320"
Sep 09 21:01:02.806970 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:02.806921    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f44688b45-2nrvb" podStartSLOduration=1.731815003 podStartE2EDuration="4.806905243s" totalImagesPullingTime="3.07509024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:02.806043357 +0000 UTC m=+751.308873359" watchObservedRunningTime="2026-09-09 21:01:02.806905243 +0000 UTC m=+751.309735292"
Sep 09 21:01:13.774479 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:13.774446    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:01:13.774868 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:13.774529    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:01:13.851304 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:13.851260    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-868c87cfb8-7rd89"]
Sep 09 21:01:13.851550 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:13.851487    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-868c87cfb8-7rd89" podUID="d38d8b43-901d-46c6-86a7-2034e6d1e989" containerName="manager" containerID="cri-o://600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e" gracePeriod=30
Sep 09 21:01:14.079276 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.079254    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:01:14.140224 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.140190    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs\") pod \"d38d8b43-901d-46c6-86a7-2034e6d1e989\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") "
Sep 09 21:01:14.140338 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.140261    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert\") pod \"d38d8b43-901d-46c6-86a7-2034e6d1e989\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") "
Sep 09 21:01:14.140338 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.140317    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqdm5\" (UniqueName: \"kubernetes.io/projected/d38d8b43-901d-46c6-86a7-2034e6d1e989-kube-api-access-sqdm5\") pod \"d38d8b43-901d-46c6-86a7-2034e6d1e989\" (UID: \"d38d8b43-901d-46c6-86a7-2034e6d1e989\") "
Sep 09 21:01:14.142370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.142344    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d38d8b43-901d-46c6-86a7-2034e6d1e989" (UID: "d38d8b43-901d-46c6-86a7-2034e6d1e989"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:01:14.142370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.142349    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38d8b43-901d-46c6-86a7-2034e6d1e989-kube-api-access-sqdm5" (OuterVolumeSpecName: "kube-api-access-sqdm5") pod "d38d8b43-901d-46c6-86a7-2034e6d1e989" (UID: "d38d8b43-901d-46c6-86a7-2034e6d1e989"). InnerVolumeSpecName "kube-api-access-sqdm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:14.142502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.142372    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d38d8b43-901d-46c6-86a7-2034e6d1e989" (UID: "d38d8b43-901d-46c6-86a7-2034e6d1e989"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:01:14.240906 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.240879    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqdm5\" (UniqueName: \"kubernetes.io/projected/d38d8b43-901d-46c6-86a7-2034e6d1e989-kube-api-access-sqdm5\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:01:14.240906 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.240903    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:01:14.241040 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.240912    2548 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d38d8b43-901d-46c6-86a7-2034e6d1e989-webhook-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:01:14.832155 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.832122    2548 generic.go:356] "Generic (PLEG): container finished" podID="d38d8b43-901d-46c6-86a7-2034e6d1e989" containerID="600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e" exitCode=0
Sep 09 21:01:14.832529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.832169    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868c87cfb8-7rd89"
Sep 09 21:01:14.832529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.832193    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868c87cfb8-7rd89" event={"ID":"d38d8b43-901d-46c6-86a7-2034e6d1e989","Type":"ContainerDied","Data":"600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e"}
Sep 09 21:01:14.832529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.832236    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868c87cfb8-7rd89" event={"ID":"d38d8b43-901d-46c6-86a7-2034e6d1e989","Type":"ContainerDied","Data":"d7994fe18955989f4b7b6981e74ffdeb3b5e685c27ffabe220e678d7af4b2992"}
Sep 09 21:01:14.832529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.832256    2548 scope.go:117] "RemoveContainer" containerID="600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e"
Sep 09 21:01:14.841530 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.841514    2548 scope.go:117] "RemoveContainer" containerID="600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e"
Sep 09 21:01:14.841803 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:01:14.841784    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e\": container with ID starting with 600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e not found: ID does not exist" containerID="600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e"
Sep 09 21:01:14.841865 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.841812    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e"} err="failed to get container status \"600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e\": rpc error: code = NotFound desc = could not find container \"600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e\": container with ID starting with 600776301c55363ec5ad0224e4d696a141a3833c7e09583051ddd089d766210e not found: ID does not exist"
Sep 09 21:01:14.854872 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.854845    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-868c87cfb8-7rd89"]
Sep 09 21:01:14.860587 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:14.860567    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-868c87cfb8-7rd89"]
Sep 09 21:01:15.967451 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:15.967410    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38d8b43-901d-46c6-86a7-2034e6d1e989" path="/var/lib/kubelet/pods/d38d8b43-901d-46c6-86a7-2034e6d1e989/volumes"
Sep 09 21:01:32.789596 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.789548    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-p8md7"]
Sep 09 21:01:32.790031 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.789962    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38d8b43-901d-46c6-86a7-2034e6d1e989" containerName="manager"
Sep 09 21:01:32.790031 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.789978    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38d8b43-901d-46c6-86a7-2034e6d1e989" containerName="manager"
Sep 09 21:01:32.790117 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.790034    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38d8b43-901d-46c6-86a7-2034e6d1e989" containerName="manager"
Sep 09 21:01:32.796486 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.796466    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.799353 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.799332    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-gateway-identity\""
Sep 09 21:01:32.800191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.800156    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 21:01:32.800302 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.800197    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 21:01:32.800444 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.800429    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gj4rp\""
Sep 09 21:01:32.800979 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.800960    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 21:01:32.801126 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.801006    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 21:01:32.804426 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.803575    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-p8md7"]
Sep 09 21:01:32.890524 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.890489    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.890524 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.890527    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f79f8e-ebc2-43a5-b8b5-72400aa31111-openshift-service-ca\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.890805 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.890580    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9m4\" (UniqueName: \"kubernetes.io/projected/65f79f8e-ebc2-43a5-b8b5-72400aa31111-kube-api-access-5p9m4\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.890805 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.890614    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-metrics-certs\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.991055 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.991026    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-metrics-certs\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.991226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.991080    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.991226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.991106    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f79f8e-ebc2-43a5-b8b5-72400aa31111-openshift-service-ca\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.991226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.991157    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9m4\" (UniqueName: \"kubernetes.io/projected/65f79f8e-ebc2-43a5-b8b5-72400aa31111-kube-api-access-5p9m4\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.991226 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:01:32.991193    2548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 09 21:01:32.991439 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:01:32.991267    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls podName:65f79f8e-ebc2-43a5-b8b5-72400aa31111 nodeName:}" failed. No retries permitted until 2026-09-09 21:01:33.491249666 +0000 UTC m=+781.994079651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls") pod "maas-api-68d778f44-p8md7" (UID: "65f79f8e-ebc2-43a5-b8b5-72400aa31111") : secret "maas-api-serving-cert" not found
Sep 09 21:01:32.991898 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.991877    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f79f8e-ebc2-43a5-b8b5-72400aa31111-openshift-service-ca\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:32.993874 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:32.993849    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-metrics-certs\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:33.000348 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.000322    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9m4\" (UniqueName: \"kubernetes.io/projected/65f79f8e-ebc2-43a5-b8b5-72400aa31111-kube-api-access-5p9m4\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:33.303861 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.303829    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"]
Sep 09 21:01:33.307773 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.307753    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.309860 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.309840    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4pppm\""
Sep 09 21:01:33.309954 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.309936    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 21:01:33.318760 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.318718    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"]
Sep 09 21:01:33.395146 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.395112    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f3dc36-6f84-47e2-bfd9-a60c2350fea4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-pvw2q\" (UID: \"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.395281 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.395168    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2nn\" (UniqueName: \"kubernetes.io/projected/d4f3dc36-6f84-47e2-bfd9-a60c2350fea4-kube-api-access-cs2nn\") pod \"payload-pre-processing-59fbc47bf8-pvw2q\" (UID: \"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.496445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.496410    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:33.496608 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.496513    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f3dc36-6f84-47e2-bfd9-a60c2350fea4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-pvw2q\" (UID: \"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.496608 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.496584    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2nn\" (UniqueName: \"kubernetes.io/projected/d4f3dc36-6f84-47e2-bfd9-a60c2350fea4-kube-api-access-cs2nn\") pod \"payload-pre-processing-59fbc47bf8-pvw2q\" (UID: \"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.497360 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.497326    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f3dc36-6f84-47e2-bfd9-a60c2350fea4-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-pvw2q\" (UID: \"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.499255 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.499230    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls\") pod \"maas-api-68d778f44-p8md7\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") " pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:33.507386 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.507360    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2nn\" (UniqueName: \"kubernetes.io/projected/d4f3dc36-6f84-47e2-bfd9-a60c2350fea4-kube-api-access-cs2nn\") pod \"payload-pre-processing-59fbc47bf8-pvw2q\" (UID: \"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.618316 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.618241    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:33.708612 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.708584    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:33.892542 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.892517    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"]
Sep 09 21:01:33.894213 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:01:33.894180    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f3dc36_6f84_47e2_bfd9_a60c2350fea4.slice/crio-054aa052ce33a5bc2dda3ee28b8e306796f3835ea44d17cfcf3bba2b29140a69 WatchSource:0}: Error finding container 054aa052ce33a5bc2dda3ee28b8e306796f3835ea44d17cfcf3bba2b29140a69: Status 404 returned error can't find the container with id 054aa052ce33a5bc2dda3ee28b8e306796f3835ea44d17cfcf3bba2b29140a69
Sep 09 21:01:33.921186 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.921159    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q" event={"ID":"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4","Type":"ContainerStarted","Data":"054aa052ce33a5bc2dda3ee28b8e306796f3835ea44d17cfcf3bba2b29140a69"}
Sep 09 21:01:33.983016 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:33.982993    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-p8md7"]
Sep 09 21:01:33.984712 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:01:33.984682    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f79f8e_ebc2_43a5_b8b5_72400aa31111.slice/crio-4cd5e72c3d07404faa2a506be86bb2f23339814e49f9a1a81cf52ed11e590a03 WatchSource:0}: Error finding container 4cd5e72c3d07404faa2a506be86bb2f23339814e49f9a1a81cf52ed11e590a03: Status 404 returned error can't find the container with id 4cd5e72c3d07404faa2a506be86bb2f23339814e49f9a1a81cf52ed11e590a03
Sep 09 21:01:34.930153 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:34.930075    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7" event={"ID":"65f79f8e-ebc2-43a5-b8b5-72400aa31111","Type":"ContainerStarted","Data":"4cd5e72c3d07404faa2a506be86bb2f23339814e49f9a1a81cf52ed11e590a03"}
Sep 09 21:01:37.946852 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:37.946808    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7" event={"ID":"65f79f8e-ebc2-43a5-b8b5-72400aa31111","Type":"ContainerStarted","Data":"18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15"}
Sep 09 21:01:37.947668 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:37.947641    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:37.949587 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:37.949557    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q" event={"ID":"d4f3dc36-6f84-47e2-bfd9-a60c2350fea4","Type":"ContainerStarted","Data":"a598109f4b3c1180181d2cbc044178b199bee83a893f9c069f68f33dc1a9de09"}
Sep 09 21:01:37.950089 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:37.950063    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:37.966597 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:37.966546    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7" podStartSLOduration=2.546166187 podStartE2EDuration="5.966528362s" totalImagesPullingTime="3.420362175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:37.965933143 +0000 UTC m=+786.468763145" watchObservedRunningTime="2026-09-09 21:01:37.966528362 +0000 UTC m=+786.469358363"
Sep 09 21:01:37.985492 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:37.985447    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q" podStartSLOduration=1.406488441 podStartE2EDuration="4.985433257s" totalImagesPullingTime="3.578944816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:37.983163978 +0000 UTC m=+786.485993981" watchObservedRunningTime="2026-09-09 21:01:37.985433257 +0000 UTC m=+786.488263257"
Sep 09 21:01:42.982674 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:42.982639    2548 generic.go:356] "Generic (PLEG): container finished" podID="ffd75c7e-ad26-454b-a660-e8e41929a0d5" containerID="29625935dccbf33620628fffd396df9f75789644e66de36ded1ffa16f3ad7589" exitCode=0
Sep 09 21:01:42.983102 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:42.982692    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" event={"ID":"ffd75c7e-ad26-454b-a660-e8e41929a0d5","Type":"ContainerDied","Data":"29625935dccbf33620628fffd396df9f75789644e66de36ded1ffa16f3ad7589"}
Sep 09 21:01:42.983102 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:42.983059    2548 scope.go:117] "RemoveContainer" containerID="29625935dccbf33620628fffd396df9f75789644e66de36ded1ffa16f3ad7589"
Sep 09 21:01:42.984009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:42.983979    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:42.984081 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:42.984046    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:42.984120 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:42.984078    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:43.507775 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:43.507722    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:01:43.989330 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:43.989290    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" event={"ID":"ffd75c7e-ad26-454b-a660-e8e41929a0d5","Type":"ContainerStarted","Data":"fde3549bb79da3828e141e2b85b77a4b45025c9e5236587cf360d04b9d9150f3"}
Sep 09 21:01:44.544170 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:44.544130    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:01:44.545307 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:44.545266    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 09 21:01:44.545452 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:44.545329    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" podUID="ffd75c7e-ad26-454b-a660-e8e41929a0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 09 21:01:44.973195 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:44.973160    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:01:45.544237 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:45.544201    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 09 21:01:45.544627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:45.544254    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" podUID="ffd75c7e-ad26-454b-a660-e8e41929a0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 09 21:01:46.544494 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:46.544460    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 09 21:01:46.544955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:46.544530    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" podUID="ffd75c7e-ad26-454b-a660-e8e41929a0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 09 21:01:47.543855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:47.543815    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 09 21:01:47.544059 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:47.543875    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" podUID="ffd75c7e-ad26-454b-a660-e8e41929a0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 09 21:01:48.543716 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:48.543676    2548 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nrds4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 09 21:01:48.544116 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:48.543761    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4" podUID="ffd75c7e-ad26-454b-a660-e8e41929a0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 09 21:01:49.547757 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:49.547707    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:01:49.548179 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:49.548150    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:01:49.548825 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:49.548807    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nrds4"
Sep 09 21:01:49.968622 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:49.968589    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-pvw2q"
Sep 09 21:01:59.618490 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.618397    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"]
Sep 09 21:01:59.622164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.622139    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.624323 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.624299    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mnpqm\""
Sep 09 21:01:59.624455 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.624436    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 21:01:59.624986 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.624971    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 21:01:59.625064 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.625025    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 21:01:59.633392 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.633369    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"]
Sep 09 21:01:59.738397 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.738368    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065cb0cf-c562-4e61-ba16-33a90b50e89e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.738560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.738424    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcd8\" (UniqueName: \"kubernetes.io/projected/065cb0cf-c562-4e61-ba16-33a90b50e89e-kube-api-access-gtcd8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.738560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.738468    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.738560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.738520    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.738560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.738535    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.738747 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.738561    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839406 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839369    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839569 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839415    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839569 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839462    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839569 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839535    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065cb0cf-c562-4e61-ba16-33a90b50e89e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839718 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839569    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcd8\" (UniqueName: \"kubernetes.io/projected/065cb0cf-c562-4e61-ba16-33a90b50e89e-kube-api-access-gtcd8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839718 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839625    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.839934 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839902    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.840014 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.839954    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.840074 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.840043    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.841643 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.841613    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/065cb0cf-c562-4e61-ba16-33a90b50e89e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.842080 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.842057    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/065cb0cf-c562-4e61-ba16-33a90b50e89e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.852021 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.851996    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcd8\" (UniqueName: \"kubernetes.io/projected/065cb0cf-c562-4e61-ba16-33a90b50e89e-kube-api-access-gtcd8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl\" (UID: \"065cb0cf-c562-4e61-ba16-33a90b50e89e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:01:59.933115 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:01:59.933016    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:02:00.211962 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:00.211764    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"]
Sep 09 21:02:00.214012 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:02:00.213988    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065cb0cf_c562_4e61_ba16_33a90b50e89e.slice/crio-ced749eb5787b46e776993e2c55f4a2af6de3c51088f8ec48987c0e56617b491 WatchSource:0}: Error finding container ced749eb5787b46e776993e2c55f4a2af6de3c51088f8ec48987c0e56617b491: Status 404 returned error can't find the container with id ced749eb5787b46e776993e2c55f4a2af6de3c51088f8ec48987c0e56617b491
Sep 09 21:02:01.073365 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:01.073282    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl" event={"ID":"065cb0cf-c562-4e61-ba16-33a90b50e89e","Type":"ContainerStarted","Data":"ced749eb5787b46e776993e2c55f4a2af6de3c51088f8ec48987c0e56617b491"}
Sep 09 21:02:02.080374 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:02.080328    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl" event={"ID":"065cb0cf-c562-4e61-ba16-33a90b50e89e","Type":"ContainerStarted","Data":"2cd8e3fb09e678fb15c3240985986e113ea40c0604e44072e0b16124162a9743"}
Sep 09 21:02:02.104641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:02.104591    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl" podStartSLOduration=2.080473679 podStartE2EDuration="3.104574549s" totalImagesPullingTime="1.02410087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:02.102641486 +0000 UTC m=+810.605471488" watchObservedRunningTime="2026-09-09 21:02:02.104574549 +0000 UTC m=+810.607404550"
Sep 09 21:02:09.933826 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:09.933789    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:02:09.934288 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:09.933901    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:02:09.945850 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:09.945827    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:02:10.129818 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:10.129785    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl"
Sep 09 21:02:14.440131 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.440097    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"]
Sep 09 21:02:14.444880 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.444851    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.447908 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.447879    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 09 21:02:14.466226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.466197    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"]
Sep 09 21:02:14.469467 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.469443    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.469605 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.469504    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.469605 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.469547    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvc7\" (UniqueName: \"kubernetes.io/projected/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-kube-api-access-pcvc7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.469605 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.469595    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.469795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.469635    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.469795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.469782    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.571069 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.571022    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvc7\" (UniqueName: \"kubernetes.io/projected/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-kube-api-access-pcvc7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.571625 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.571590    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.571779 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.571635    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.571779 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.571699    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.571909 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.571776    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.571909 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.571834    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.572103 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.572079    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.572435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.572406    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.572549 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.572511    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.574330 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.574305    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.574897 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.574876    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.581803 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.581778    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvc7\" (UniqueName: \"kubernetes.io/projected/8bbe6b08-eef5-43f6-a921-d71ed0aa1b55-kube-api-access-pcvc7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-khlwj\" (UID: \"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:14.756059 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:14.755980    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:15.028609 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:15.028583    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"]
Sep 09 21:02:15.030304 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:02:15.030275    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbe6b08_eef5_43f6_a921_d71ed0aa1b55.slice/crio-c961656b3f81edb937f6c5b8901534cab05cc87528de16451d4ee0e4c2a7d80e WatchSource:0}: Error finding container c961656b3f81edb937f6c5b8901534cab05cc87528de16451d4ee0e4c2a7d80e: Status 404 returned error can't find the container with id c961656b3f81edb937f6c5b8901534cab05cc87528de16451d4ee0e4c2a7d80e
Sep 09 21:02:15.142771 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:15.142723    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj" event={"ID":"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55","Type":"ContainerStarted","Data":"c961656b3f81edb937f6c5b8901534cab05cc87528de16451d4ee0e4c2a7d80e"}
Sep 09 21:02:16.150193 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:16.150156    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj" event={"ID":"8bbe6b08-eef5-43f6-a921-d71ed0aa1b55","Type":"ContainerStarted","Data":"e5970689aff835d9bce712fac1326bb3fa9b9ece1a056b9037ddbe004c473fb5"}
Sep 09 21:02:16.168442 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:16.168388    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj" podStartSLOduration=1.959799388 podStartE2EDuration="2.168368965s" totalImagesPullingTime="208.569577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:16.165798765 +0000 UTC m=+824.668628766" watchObservedRunningTime="2026-09-09 21:02:16.168368965 +0000 UTC m=+824.671198971"
Sep 09 21:02:24.757126 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:24.757087    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:24.757583 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:24.757159    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:24.769447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:24.769424    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:25.203213 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:25.203188    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-khlwj"
Sep 09 21:02:59.207462 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.207425    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"]
Sep 09 21:02:59.211883 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.211863    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.214402 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.214377    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 21:02:59.221060 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.220593    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"]
Sep 09 21:02:59.261961 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.261918    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.261961 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.261962    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.262174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.262063    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.262174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.262148    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkngx\" (UniqueName: \"kubernetes.io/projected/26f3d97f-7318-407b-adbf-83b5b9eb4598-kube-api-access-jkngx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.262259 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.262191    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.262300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.262269    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f3d97f-7318-407b-adbf-83b5b9eb4598-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363298 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363255    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkngx\" (UniqueName: \"kubernetes.io/projected/26f3d97f-7318-407b-adbf-83b5b9eb4598-kube-api-access-jkngx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363318    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363381    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f3d97f-7318-407b-adbf-83b5b9eb4598-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363402    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363423    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363465    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363721    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363791    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.363972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.363857    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.365750 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.365682    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f3d97f-7318-407b-adbf-83b5b9eb4598-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.366018 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.365996    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f3d97f-7318-407b-adbf-83b5b9eb4598-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.370658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.370634    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkngx\" (UniqueName: \"kubernetes.io/projected/26f3d97f-7318-407b-adbf-83b5b9eb4598-kube-api-access-jkngx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt\" (UID: \"26f3d97f-7318-407b-adbf-83b5b9eb4598\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.524484 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.524386    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:02:59.809673 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:02:59.809638    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"]
Sep 09 21:02:59.811406 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:02:59.811376    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26f3d97f_7318_407b_adbf_83b5b9eb4598.slice/crio-be5b29710749c5e757df5fd53b920829fae398e223c246f4c1c6c98fe1c6197f WatchSource:0}: Error finding container be5b29710749c5e757df5fd53b920829fae398e223c246f4c1c6c98fe1c6197f: Status 404 returned error can't find the container with id be5b29710749c5e757df5fd53b920829fae398e223c246f4c1c6c98fe1c6197f
Sep 09 21:03:00.363947 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:00.363847    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt" event={"ID":"26f3d97f-7318-407b-adbf-83b5b9eb4598","Type":"ContainerStarted","Data":"26a729225b2a5f08f4d3f15369649a249f810028df3a9df972d0c2b16c80c5f8"}
Sep 09 21:03:00.363947 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:00.363891    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt" event={"ID":"26f3d97f-7318-407b-adbf-83b5b9eb4598","Type":"ContainerStarted","Data":"be5b29710749c5e757df5fd53b920829fae398e223c246f4c1c6c98fe1c6197f"}
Sep 09 21:03:00.396761 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:00.396695    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt" podStartSLOduration=1.154523486 podStartE2EDuration="1.396680079s" totalImagesPullingTime="242.156593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:00.395526223 +0000 UTC m=+868.898356237" watchObservedRunningTime="2026-09-09 21:03:00.396680079 +0000 UTC m=+868.899510079"
Sep 09 21:03:09.525117 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:09.525084    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:03:09.525503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:09.525128    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:03:09.538063 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:09.538041    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:03:10.422855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:10.422769    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt"
Sep 09 21:03:31.977205 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:31.977136    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:03:31.979716 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:03:31.979691    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:04:08.925110 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:08.925077    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dd5d6df4f-pm9zg"]
Sep 09 21:04:08.929074 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:08.929049    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:08.931957 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:08.931930    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8kvtd\""
Sep 09 21:04:08.932063 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:08.931991    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 21:04:08.938909 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:08.938884    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dd5d6df4f-pm9zg"]
Sep 09 21:04:09.003119 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.003089    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b5fbd73-9b5a-47f2-accc-c0c9c22c841c-tls-cert\") pod \"authorino-dd5d6df4f-pm9zg\" (UID: \"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c\") " pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.003244 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.003156    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdj8\" (UniqueName: \"kubernetes.io/projected/8b5fbd73-9b5a-47f2-accc-c0c9c22c841c-kube-api-access-btdj8\") pod \"authorino-dd5d6df4f-pm9zg\" (UID: \"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c\") " pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.104490 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.104457    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b5fbd73-9b5a-47f2-accc-c0c9c22c841c-tls-cert\") pod \"authorino-dd5d6df4f-pm9zg\" (UID: \"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c\") " pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.104630 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.104540    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdj8\" (UniqueName: \"kubernetes.io/projected/8b5fbd73-9b5a-47f2-accc-c0c9c22c841c-kube-api-access-btdj8\") pod \"authorino-dd5d6df4f-pm9zg\" (UID: \"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c\") " pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.106943 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.106922    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b5fbd73-9b5a-47f2-accc-c0c9c22c841c-tls-cert\") pod \"authorino-dd5d6df4f-pm9zg\" (UID: \"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c\") " pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.112481 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.112455    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdj8\" (UniqueName: \"kubernetes.io/projected/8b5fbd73-9b5a-47f2-accc-c0c9c22c841c-kube-api-access-btdj8\") pod \"authorino-dd5d6df4f-pm9zg\" (UID: \"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c\") " pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.238972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.238948    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd5d6df4f-pm9zg"
Sep 09 21:04:09.507224 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.507201    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dd5d6df4f-pm9zg"]
Sep 09 21:04:09.509437 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:04:09.509410    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b5fbd73_9b5a_47f2_accc_c0c9c22c841c.slice/crio-5ab6538c8155000cf5eae22291f96e77cbc9de39e717aecf17414db9be04538a WatchSource:0}: Error finding container 5ab6538c8155000cf5eae22291f96e77cbc9de39e717aecf17414db9be04538a: Status 404 returned error can't find the container with id 5ab6538c8155000cf5eae22291f96e77cbc9de39e717aecf17414db9be04538a
Sep 09 21:04:09.701805 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:09.701771    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd5d6df4f-pm9zg" event={"ID":"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c","Type":"ContainerStarted","Data":"5ab6538c8155000cf5eae22291f96e77cbc9de39e717aecf17414db9be04538a"}
Sep 09 21:04:12.718582 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:12.718540    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd5d6df4f-pm9zg" event={"ID":"8b5fbd73-9b5a-47f2-accc-c0c9c22c841c","Type":"ContainerStarted","Data":"e41c0c898d297f468ca70ef16dadfda112bea3fbf03b388623fd47fb101c0d66"}
Sep 09 21:04:12.739715 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:12.739663    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-dd5d6df4f-pm9zg" podStartSLOduration=2.463774841 podStartE2EDuration="4.739649305s" totalImagesPullingTime="2.275874464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:12.737773676 +0000 UTC m=+941.240603678" watchObservedRunningTime="2026-09-09 21:04:12.739649305 +0000 UTC m=+941.242479305"
Sep 09 21:04:29.945620 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.945583    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"]
Sep 09 21:04:29.949988 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.949961    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.952898 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.952873    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9a54ea-openshift-default-dockercfg-zklw6\""
Sep 09 21:04:29.995173 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995136    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-socket\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995173 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995173    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-credential-socket\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995334 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995190    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-istio-token\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995334 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995208    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310a117b-3658-4f4a-82e2-539629e0cfad-istiod-ca-cert\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995334 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995284    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-envoy\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995334 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995308    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-data\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995484 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995338    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6kd\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-kube-api-access-fg6kd\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995484 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995358    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310a117b-3658-4f4a-82e2-539629e0cfad-istio-podinfo\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:29.995484 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:29.995396    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-certs\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.012831 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.012769    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"]
Sep 09 21:04:30.096842 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.096809    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310a117b-3658-4f4a-82e2-539629e0cfad-istio-podinfo\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097021 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.096871    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-certs\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097021 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097013    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-socket\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097046    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-credential-socket\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097073    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-istio-token\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097103    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310a117b-3658-4f4a-82e2-539629e0cfad-istiod-ca-cert\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097133    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-envoy\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097330 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097156    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-data\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097330 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097199    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6kd\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-kube-api-access-fg6kd\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097428 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097348    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-certs\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097428 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097358    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-socket\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097607 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097579    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-credential-socket\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097706 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097636    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-data\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.097931 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.097908    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310a117b-3658-4f4a-82e2-539629e0cfad-istiod-ca-cert\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.100027 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.100005    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-envoy\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.100146 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.100128    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310a117b-3658-4f4a-82e2-539629e0cfad-istio-podinfo\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.111131 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.111101    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-istio-token\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.111324 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.111308    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6kd\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-kube-api-access-fg6kd\") pod \"e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") " pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.261337 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.261266    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:30.545249 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.545165    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"]
Sep 09 21:04:30.549355 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:04:30.549320    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310a117b_3658_4f4a_82e2_539629e0cfad.slice/crio-0ad0f3585eb1a6a0c3da3c82e814f424ad32d52139e94d2b9e6bba978068e8f0 WatchSource:0}: Error finding container 0ad0f3585eb1a6a0c3da3c82e814f424ad32d52139e94d2b9e6bba978068e8f0: Status 404 returned error can't find the container with id 0ad0f3585eb1a6a0c3da3c82e814f424ad32d52139e94d2b9e6bba978068e8f0
Sep 09 21:04:30.551657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.551618    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:04:30.551759 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.551719    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:04:30.551808 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.551780    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:04:30.806539 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.806454    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" event={"ID":"310a117b-3658-4f4a-82e2-539629e0cfad","Type":"ContainerStarted","Data":"d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"}
Sep 09 21:04:30.806539 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.806526    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" event={"ID":"310a117b-3658-4f4a-82e2-539629e0cfad","Type":"ContainerStarted","Data":"0ad0f3585eb1a6a0c3da3c82e814f424ad32d52139e94d2b9e6bba978068e8f0"}
Sep 09 21:04:30.828069 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:30.828018    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podStartSLOduration=1.828003135 podStartE2EDuration="1.828003135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:30.82677351 +0000 UTC m=+959.329603510" watchObservedRunningTime="2026-09-09 21:04:30.828003135 +0000 UTC m=+959.330833137"
Sep 09 21:04:31.262490 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:31.262457    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:31.267292 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:31.267270    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:31.811867 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:31.811835    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:31.812617 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:31.812599    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:34.802465 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.802430    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"]
Sep 09 21:04:34.806150 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.806126    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:34.808504 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.808480    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9a54ea-dockercfg-9h9j9\""
Sep 09 21:04:34.808717 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.808500    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9a54ea\""
Sep 09 21:04:34.817852 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.817828    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"]
Sep 09 21:04:34.848483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.846295    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858be8e1-49e0-41a4-8ad1-0023d993bdfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:34.848483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.846384    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwp6\" (UniqueName: \"kubernetes.io/projected/858be8e1-49e0-41a4-8ad1-0023d993bdfb-kube-api-access-vzwp6\") pod \"payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:34.880196 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.880164    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"]
Sep 09 21:04:34.883682 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.883661    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:34.895456 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.895114    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"]
Sep 09 21:04:34.947344 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.947310    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jbh\" (UniqueName: \"kubernetes.io/projected/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-kube-api-access-g9jbh\") pod \"payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:34.947636 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.947616    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858be8e1-49e0-41a4-8ad1-0023d993bdfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:34.948600 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.948491    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858be8e1-49e0-41a4-8ad1-0023d993bdfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:34.948808 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.948789    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:34.948918 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.948894    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwp6\" (UniqueName: \"kubernetes.io/projected/858be8e1-49e0-41a4-8ad1-0023d993bdfb-kube-api-access-vzwp6\") pod \"payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:34.958384 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:34.958300    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwp6\" (UniqueName: \"kubernetes.io/projected/858be8e1-49e0-41a4-8ad1-0023d993bdfb-kube-api-access-vzwp6\") pod \"payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:35.050379 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.050340    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:35.050518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.050506    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jbh\" (UniqueName: \"kubernetes.io/projected/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-kube-api-access-g9jbh\") pod \"payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:35.051179 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.051154    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:35.060705 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.060650    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jbh\" (UniqueName: \"kubernetes.io/projected/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-kube-api-access-g9jbh\") pod \"payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:35.117695 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.117668    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:35.198362 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.198160    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:35.388635 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.388356    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"]
Sep 09 21:04:35.391644 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:04:35.391612    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858be8e1_49e0_41a4_8ad1_0023d993bdfb.slice/crio-604015a9281ff008c054ee11950ada2f9d10ce79a5cb32dc5d57bc7456245713 WatchSource:0}: Error finding container 604015a9281ff008c054ee11950ada2f9d10ce79a5cb32dc5d57bc7456245713: Status 404 returned error can't find the container with id 604015a9281ff008c054ee11950ada2f9d10ce79a5cb32dc5d57bc7456245713
Sep 09 21:04:35.477788 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.477711    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"]
Sep 09 21:04:35.480238 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:04:35.480213    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa59509_85d3_4fe1_9c2c_1ccf27dd8815.slice/crio-20b6617cc16328bc153c0b63649567443526c80c41b2ebe73cf45944321a8514 WatchSource:0}: Error finding container 20b6617cc16328bc153c0b63649567443526c80c41b2ebe73cf45944321a8514: Status 404 returned error can't find the container with id 20b6617cc16328bc153c0b63649567443526c80c41b2ebe73cf45944321a8514
Sep 09 21:04:35.836576 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.836538    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb" event={"ID":"858be8e1-49e0-41a4-8ad1-0023d993bdfb","Type":"ContainerStarted","Data":"0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d"}
Sep 09 21:04:35.837066 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.836599    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb" event={"ID":"858be8e1-49e0-41a4-8ad1-0023d993bdfb","Type":"ContainerStarted","Data":"604015a9281ff008c054ee11950ada2f9d10ce79a5cb32dc5d57bc7456245713"}
Sep 09 21:04:35.837066 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.836709    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:35.838614 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.838586    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44" event={"ID":"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815","Type":"ContainerStarted","Data":"9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6"}
Sep 09 21:04:35.838766 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.838636    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44" event={"ID":"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815","Type":"ContainerStarted","Data":"20b6617cc16328bc153c0b63649567443526c80c41b2ebe73cf45944321a8514"}
Sep 09 21:04:35.838819 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.838761    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:35.856513 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.856461    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb" podStartSLOduration=1.856444899 podStartE2EDuration="1.856444899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:35.855346161 +0000 UTC m=+964.358176162" watchObservedRunningTime="2026-09-09 21:04:35.856444899 +0000 UTC m=+964.359274901"
Sep 09 21:04:35.876079 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:35.876032    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44" podStartSLOduration=1.876016689 podStartE2EDuration="1.876016689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:35.874584719 +0000 UTC m=+964.377414723" watchObservedRunningTime="2026-09-09 21:04:35.876016689 +0000 UTC m=+964.378846725"
Sep 09 21:04:46.076565 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.076510    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"]
Sep 09 21:04:46.082655 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.082624    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.087847 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.087824    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ab334d49\""
Sep 09 21:04:46.088107 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.088090    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ab334d49-dockercfg-8qmpm\""
Sep 09 21:04:46.108503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.108471    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"]
Sep 09 21:04:46.178415 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.178372    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"]
Sep 09 21:04:46.182792 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.182772    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.183789 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.183765    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5f5t\" (UniqueName: \"kubernetes.io/projected/30bd8d0f-92e1-42dd-8019-c5edad49c78a-kube-api-access-d5f5t\") pod \"payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.183976 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.183827    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bd8d0f-92e1-42dd-8019-c5edad49c78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.194280 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.194244    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"]
Sep 09 21:04:46.285371 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.285333    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5f5t\" (UniqueName: \"kubernetes.io/projected/30bd8d0f-92e1-42dd-8019-c5edad49c78a-kube-api-access-d5f5t\") pod \"payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.285532 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.285380    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67622\" (UniqueName: \"kubernetes.io/projected/0fbe527d-3c23-4341-97b0-521f538eb06c-kube-api-access-67622\") pod \"payload-processing-e2e-ait-ab334d49-799789bf75-vjljf\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") " pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.285532 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.285492    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bd8d0f-92e1-42dd-8019-c5edad49c78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.285605 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.285536    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbe527d-3c23-4341-97b0-521f538eb06c-plugins-config-volume\") pod \"payload-processing-e2e-ait-ab334d49-799789bf75-vjljf\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") " pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.286313 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.286282    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bd8d0f-92e1-42dd-8019-c5edad49c78a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.293147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.293119    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5f5t\" (UniqueName: \"kubernetes.io/projected/30bd8d0f-92e1-42dd-8019-c5edad49c78a-kube-api-access-d5f5t\") pod \"payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.386853 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.386761    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbe527d-3c23-4341-97b0-521f538eb06c-plugins-config-volume\") pod \"payload-processing-e2e-ait-ab334d49-799789bf75-vjljf\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") " pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.387020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.386894    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67622\" (UniqueName: \"kubernetes.io/projected/0fbe527d-3c23-4341-97b0-521f538eb06c-kube-api-access-67622\") pod \"payload-processing-e2e-ait-ab334d49-799789bf75-vjljf\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") " pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.387525 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.387502    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbe527d-3c23-4341-97b0-521f538eb06c-plugins-config-volume\") pod \"payload-processing-e2e-ait-ab334d49-799789bf75-vjljf\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") " pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.395302 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.395271    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67622\" (UniqueName: \"kubernetes.io/projected/0fbe527d-3c23-4341-97b0-521f538eb06c-kube-api-access-67622\") pod \"payload-processing-e2e-ait-ab334d49-799789bf75-vjljf\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") " pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.395422 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.395408    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.498621 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.498590    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.695832 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.695807    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"]
Sep 09 21:04:46.697812 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:04:46.697782    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bd8d0f_92e1_42dd_8019_c5edad49c78a.slice/crio-73f00e3aa381483f478fd74cefa942c3f3b08f595faa7ccb93ea575291188719 WatchSource:0}: Error finding container 73f00e3aa381483f478fd74cefa942c3f3b08f595faa7ccb93ea575291188719: Status 404 returned error can't find the container with id 73f00e3aa381483f478fd74cefa942c3f3b08f595faa7ccb93ea575291188719
Sep 09 21:04:46.813950 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.813912    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"]
Sep 09 21:04:46.815834 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:04:46.815806    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbe527d_3c23_4341_97b0_521f538eb06c.slice/crio-9f3e906e2a6d3647160a021d11a7e12b1dabd5881fd7432c4b526d967456a02a WatchSource:0}: Error finding container 9f3e906e2a6d3647160a021d11a7e12b1dabd5881fd7432c4b526d967456a02a: Status 404 returned error can't find the container with id 9f3e906e2a6d3647160a021d11a7e12b1dabd5881fd7432c4b526d967456a02a
Sep 09 21:04:46.850625 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.850544    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:04:46.852723 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.852091    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:04:46.918208 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.918071    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf" event={"ID":"0fbe527d-3c23-4341-97b0-521f538eb06c","Type":"ContainerStarted","Data":"fa4ed3e512182801fc2427c59cbe2d738d567b4c43a904dc12853fb215f140c9"}
Sep 09 21:04:46.918208 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.918141    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf" event={"ID":"0fbe527d-3c23-4341-97b0-521f538eb06c","Type":"ContainerStarted","Data":"9f3e906e2a6d3647160a021d11a7e12b1dabd5881fd7432c4b526d967456a02a"}
Sep 09 21:04:46.918447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.918426    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:04:46.920802 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.920507    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g" event={"ID":"30bd8d0f-92e1-42dd-8019-c5edad49c78a","Type":"ContainerStarted","Data":"0eea8a576ec02aed178dfda8e9dc8a8e56903b5d4eebc77e12df9120f1cbda82"}
Sep 09 21:04:46.920802 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.920557    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g" event={"ID":"30bd8d0f-92e1-42dd-8019-c5edad49c78a","Type":"ContainerStarted","Data":"73f00e3aa381483f478fd74cefa942c3f3b08f595faa7ccb93ea575291188719"}
Sep 09 21:04:46.921012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.920901    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:46.938515 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.938458    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf" podStartSLOduration=0.938440431 podStartE2EDuration="938.440431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:46.936115139 +0000 UTC m=+975.438945141" watchObservedRunningTime="2026-09-09 21:04:46.938440431 +0000 UTC m=+975.441270433"
Sep 09 21:04:46.956133 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:46.955378    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g" podStartSLOduration=0.955359539 podStartE2EDuration="955.359539ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:46.952796543 +0000 UTC m=+975.455626545" watchObservedRunningTime="2026-09-09 21:04:46.955359539 +0000 UTC m=+975.458189541"
Sep 09 21:04:50.945982 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:50.945943    2548 generic.go:356] "Generic (PLEG): container finished" podID="310a117b-3658-4f4a-82e2-539629e0cfad" containerID="d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0" exitCode=0
Sep 09 21:04:50.946485 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:50.945994    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" event={"ID":"310a117b-3658-4f4a-82e2-539629e0cfad","Type":"ContainerDied","Data":"d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"}
Sep 09 21:04:50.946485 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:50.946381    2548 scope.go:117] "RemoveContainer" containerID="d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"
Sep 09 21:04:50.947530 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:50.947496    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:04:50.947652 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:50.947586    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:04:50.947652 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:50.947637    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:04:51.956644 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:51.956606    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" event={"ID":"310a117b-3658-4f4a-82e2-539629e0cfad","Type":"ContainerStarted","Data":"b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a"}
Sep 09 21:04:52.262549 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:52.262452    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:52.262716 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:52.262596    2548 patch_prober.go:28] interesting pod/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 09 21:04:52.262716 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:52.262640    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 09 21:04:53.263051 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:53.263003    2548 patch_prober.go:28] interesting pod/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 09 21:04:53.263533 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:53.263087    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 09 21:04:54.262300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:54.262253    2548 patch_prober.go:28] interesting pod/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 09 21:04:54.262503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:54.262326    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 09 21:04:55.262669 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:55.262626    2548 patch_prober.go:28] interesting pod/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 09 21:04:55.263161 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:55.262698    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 09 21:04:56.262707 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:56.262666    2548 patch_prober.go:28] interesting pod/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 09 21:04:56.263333 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:56.263297    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 09 21:04:57.273849 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:57.273810    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:57.274275 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:57.274087    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:57.274690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:57.274672    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:04:57.929598 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:57.929562    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:04:57.929859 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:04:57.929840    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:05:02.603386 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.603342    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"]
Sep 09 21:05:02.607441 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.607417    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.609957 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.609933    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-98dad6-serving-cert\""
Sep 09 21:05:02.619260 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.619231    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"]
Sep 09 21:05:02.668273 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.668226    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.668445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.668341    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-metrics-certs\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.668445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.668437    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543dd57a-99f1-4090-9247-69420faa2b95-openshift-service-ca\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.668527 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.668494    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb256\" (UniqueName: \"kubernetes.io/projected/543dd57a-99f1-4090-9247-69420faa2b95-kube-api-access-tb256\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.769590 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.769546    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb256\" (UniqueName: \"kubernetes.io/projected/543dd57a-99f1-4090-9247-69420faa2b95-kube-api-access-tb256\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.769827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.769605    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.769827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.769661    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-metrics-certs\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.769827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.769771    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543dd57a-99f1-4090-9247-69420faa2b95-openshift-service-ca\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.769827 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:05:02.769809    2548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-serving-cert: secret "maas-api-e2e-inf-b-98dad6-serving-cert" not found
Sep 09 21:05:02.770023 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:05:02.769879    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls podName:543dd57a-99f1-4090-9247-69420faa2b95 nodeName:}" failed. No retries permitted until 2026-09-09 21:05:03.269858262 +0000 UTC m=+991.772688251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls") pod "maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" (UID: "543dd57a-99f1-4090-9247-69420faa2b95") : secret "maas-api-e2e-inf-b-98dad6-serving-cert" not found
Sep 09 21:05:02.770487 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.770452    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543dd57a-99f1-4090-9247-69420faa2b95-openshift-service-ca\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.772187 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.772167    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-metrics-certs\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:02.785882 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:02.785857    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb256\" (UniqueName: \"kubernetes.io/projected/543dd57a-99f1-4090-9247-69420faa2b95-kube-api-access-tb256\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:03.274822 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:03.274783    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:03.277483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:03.277454    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls\") pod \"maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:03.523285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:03.523247    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:03.802185 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:03.802160    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"]
Sep 09 21:05:03.803844 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:05:03.803808    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543dd57a_99f1_4090_9247_69420faa2b95.slice/crio-8de7fdac9250c166a344401b18f1209ef79f4c6e60eba84f40d117c65ba6f53c WatchSource:0}: Error finding container 8de7fdac9250c166a344401b18f1209ef79f4c6e60eba84f40d117c65ba6f53c: Status 404 returned error can't find the container with id 8de7fdac9250c166a344401b18f1209ef79f4c6e60eba84f40d117c65ba6f53c
Sep 09 21:05:03.805460 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:03.805442    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:05:04.034676 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:04.034636    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" event={"ID":"543dd57a-99f1-4090-9247-69420faa2b95","Type":"ContainerStarted","Data":"8de7fdac9250c166a344401b18f1209ef79f4c6e60eba84f40d117c65ba6f53c"}
Sep 09 21:05:05.042565 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:05.042520    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" event={"ID":"543dd57a-99f1-4090-9247-69420faa2b95","Type":"ContainerStarted","Data":"079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8"}
Sep 09 21:05:05.042965 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:05.042594    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:05.062900 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:05.062838    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" podStartSLOduration=2.6554224140000002 podStartE2EDuration="3.062816432s" totalImagesPullingTime="407.394018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:05.060404729 +0000 UTC m=+993.563234746" watchObservedRunningTime="2026-09-09 21:05:05.062816432 +0000 UTC m=+993.565646433"
Sep 09 21:05:11.055424 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:11.055386    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:05:25.939849 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:25.939810    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"]
Sep 09 21:05:25.940457 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:25.940158    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf" podUID="0fbe527d-3c23-4341-97b0-521f538eb06c" containerName="payload-processing" containerID="cri-o://fa4ed3e512182801fc2427c59cbe2d738d567b4c43a904dc12853fb215f140c9" gracePeriod=30
Sep 09 21:05:25.942297 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:25.942251    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"]
Sep 09 21:05:25.943064 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:25.942547    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g" podUID="30bd8d0f-92e1-42dd-8019-c5edad49c78a" containerName="payload-pre-processing" containerID="cri-o://0eea8a576ec02aed178dfda8e9dc8a8e56903b5d4eebc77e12df9120f1cbda82" gracePeriod=30
Sep 09 21:05:26.169692 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.169244    2548 generic.go:356] "Generic (PLEG): container finished" podID="0fbe527d-3c23-4341-97b0-521f538eb06c" containerID="fa4ed3e512182801fc2427c59cbe2d738d567b4c43a904dc12853fb215f140c9" exitCode=0
Sep 09 21:05:26.169692 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.169442    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf" event={"ID":"0fbe527d-3c23-4341-97b0-521f538eb06c","Type":"ContainerDied","Data":"fa4ed3e512182801fc2427c59cbe2d738d567b4c43a904dc12853fb215f140c9"}
Sep 09 21:05:26.171724 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.171619    2548 generic.go:356] "Generic (PLEG): container finished" podID="30bd8d0f-92e1-42dd-8019-c5edad49c78a" containerID="0eea8a576ec02aed178dfda8e9dc8a8e56903b5d4eebc77e12df9120f1cbda82" exitCode=0
Sep 09 21:05:26.171724 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.171703    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g" event={"ID":"30bd8d0f-92e1-42dd-8019-c5edad49c78a","Type":"ContainerDied","Data":"0eea8a576ec02aed178dfda8e9dc8a8e56903b5d4eebc77e12df9120f1cbda82"}
Sep 09 21:05:26.251406 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.251369    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:05:26.255450 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.255427    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:05:26.315342 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.315270    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67622\" (UniqueName: \"kubernetes.io/projected/0fbe527d-3c23-4341-97b0-521f538eb06c-kube-api-access-67622\") pod \"0fbe527d-3c23-4341-97b0-521f538eb06c\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") "
Sep 09 21:05:26.315342 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.315344    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbe527d-3c23-4341-97b0-521f538eb06c-plugins-config-volume\") pod \"0fbe527d-3c23-4341-97b0-521f538eb06c\" (UID: \"0fbe527d-3c23-4341-97b0-521f538eb06c\") "
Sep 09 21:05:26.315593 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.315412    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5f5t\" (UniqueName: \"kubernetes.io/projected/30bd8d0f-92e1-42dd-8019-c5edad49c78a-kube-api-access-d5f5t\") pod \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") "
Sep 09 21:05:26.316380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.315879    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bd8d0f-92e1-42dd-8019-c5edad49c78a-plugins-config-volume\") pod \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\" (UID: \"30bd8d0f-92e1-42dd-8019-c5edad49c78a\") "
Sep 09 21:05:26.316380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.316260    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbe527d-3c23-4341-97b0-521f538eb06c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fbe527d-3c23-4341-97b0-521f538eb06c" (UID: "0fbe527d-3c23-4341-97b0-521f538eb06c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:26.316630 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.316604    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bd8d0f-92e1-42dd-8019-c5edad49c78a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30bd8d0f-92e1-42dd-8019-c5edad49c78a" (UID: "30bd8d0f-92e1-42dd-8019-c5edad49c78a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:26.319330 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.319299    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbe527d-3c23-4341-97b0-521f538eb06c-kube-api-access-67622" (OuterVolumeSpecName: "kube-api-access-67622") pod "0fbe527d-3c23-4341-97b0-521f538eb06c" (UID: "0fbe527d-3c23-4341-97b0-521f538eb06c"). InnerVolumeSpecName "kube-api-access-67622". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:26.320452 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.320425    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bd8d0f-92e1-42dd-8019-c5edad49c78a-kube-api-access-d5f5t" (OuterVolumeSpecName: "kube-api-access-d5f5t") pod "30bd8d0f-92e1-42dd-8019-c5edad49c78a" (UID: "30bd8d0f-92e1-42dd-8019-c5edad49c78a"). InnerVolumeSpecName "kube-api-access-d5f5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:26.417275 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.417243    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5f5t\" (UniqueName: \"kubernetes.io/projected/30bd8d0f-92e1-42dd-8019-c5edad49c78a-kube-api-access-d5f5t\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.417275 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.417269    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bd8d0f-92e1-42dd-8019-c5edad49c78a-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.417275 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.417279    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67622\" (UniqueName: \"kubernetes.io/projected/0fbe527d-3c23-4341-97b0-521f538eb06c-kube-api-access-67622\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:05:26.417490 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:26.417288    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fbe527d-3c23-4341-97b0-521f538eb06c-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:05:27.177894 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.177859    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"
Sep 09 21:05:27.178333 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.177896    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf" event={"ID":"0fbe527d-3c23-4341-97b0-521f538eb06c","Type":"ContainerDied","Data":"9f3e906e2a6d3647160a021d11a7e12b1dabd5881fd7432c4b526d967456a02a"}
Sep 09 21:05:27.178333 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.177959    2548 scope.go:117] "RemoveContainer" containerID="fa4ed3e512182801fc2427c59cbe2d738d567b4c43a904dc12853fb215f140c9"
Sep 09 21:05:27.179326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.179281    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g" event={"ID":"30bd8d0f-92e1-42dd-8019-c5edad49c78a","Type":"ContainerDied","Data":"73f00e3aa381483f478fd74cefa942c3f3b08f595faa7ccb93ea575291188719"}
Sep 09 21:05:27.179326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.179306    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"
Sep 09 21:05:27.189219 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.189193    2548 scope.go:117] "RemoveContainer" containerID="0eea8a576ec02aed178dfda8e9dc8a8e56903b5d4eebc77e12df9120f1cbda82"
Sep 09 21:05:27.207007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.206983    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"]
Sep 09 21:05:27.210505 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.210479    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ab334d49-799789bf75-vjljf"]
Sep 09 21:05:27.223796 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.223776    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"]
Sep 09 21:05:27.234128 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.234109    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ab334d49-5d699d9c9d-9qc5g"]
Sep 09 21:05:27.969454 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.969421    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fbe527d-3c23-4341-97b0-521f538eb06c" path="/var/lib/kubelet/pods/0fbe527d-3c23-4341-97b0-521f538eb06c/volumes"
Sep 09 21:05:27.969801 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:27.969787    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30bd8d0f-92e1-42dd-8019-c5edad49c78a" path="/var/lib/kubelet/pods/30bd8d0f-92e1-42dd-8019-c5edad49c78a/volumes"
Sep 09 21:05:30.206678 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.206644    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"]
Sep 09 21:05:30.207226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.207208    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bd8d0f-92e1-42dd-8019-c5edad49c78a" containerName="payload-pre-processing"
Sep 09 21:05:30.207274 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.207229    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bd8d0f-92e1-42dd-8019-c5edad49c78a" containerName="payload-pre-processing"
Sep 09 21:05:30.207274 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.207251    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbe527d-3c23-4341-97b0-521f538eb06c" containerName="payload-processing"
Sep 09 21:05:30.207274 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.207257    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbe527d-3c23-4341-97b0-521f538eb06c" containerName="payload-processing"
Sep 09 21:05:30.207412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.207320    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bd8d0f-92e1-42dd-8019-c5edad49c78a" containerName="payload-pre-processing"
Sep 09 21:05:30.207412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.207329    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbe527d-3c23-4341-97b0-521f538eb06c" containerName="payload-processing"
Sep 09 21:05:30.210191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.210171    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.212504 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.212486    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e62fbb04-serving-cert\""
Sep 09 21:05:30.224119 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.224097    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"]
Sep 09 21:05:30.256291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.256261    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-metrics-certs\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.256422 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.256329    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.256476 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.256439    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spw6m\" (UniqueName: \"kubernetes.io/projected/4f97f6bd-50a6-485d-b960-ebda8e5944af-kube-api-access-spw6m\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.256566 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.256539    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f97f6bd-50a6-485d-b960-ebda8e5944af-openshift-service-ca\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.357528 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.357490    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-metrics-certs\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.357685 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.357542    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.357685 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.357574    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spw6m\" (UniqueName: \"kubernetes.io/projected/4f97f6bd-50a6-485d-b960-ebda8e5944af-kube-api-access-spw6m\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.357685 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.357621    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f97f6bd-50a6-485d-b960-ebda8e5944af-openshift-service-ca\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.357840 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:05:30.357697    2548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-serving-cert: secret "maas-api-e2e-migrate-e62fbb04-serving-cert" not found
Sep 09 21:05:30.357840 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:05:30.357804    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls podName:4f97f6bd-50a6-485d-b960-ebda8e5944af nodeName:}" failed. No retries permitted until 2026-09-09 21:05:30.857782897 +0000 UTC m=+1019.360612896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls") pod "maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" (UID: "4f97f6bd-50a6-485d-b960-ebda8e5944af") : secret "maas-api-e2e-migrate-e62fbb04-serving-cert" not found
Sep 09 21:05:30.358266 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.358249    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f97f6bd-50a6-485d-b960-ebda8e5944af-openshift-service-ca\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.359980 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.359961    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-metrics-certs\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.367107 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.367079    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spw6m\" (UniqueName: \"kubernetes.io/projected/4f97f6bd-50a6-485d-b960-ebda8e5944af-kube-api-access-spw6m\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.863689 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.863655    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:30.863865 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:05:30.863821    2548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-serving-cert: secret "maas-api-e2e-migrate-e62fbb04-serving-cert" not found
Sep 09 21:05:30.863920 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:05:30.863902    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls podName:4f97f6bd-50a6-485d-b960-ebda8e5944af nodeName:}" failed. No retries permitted until 2026-09-09 21:05:31.863882657 +0000 UTC m=+1020.366712648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls") pod "maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" (UID: "4f97f6bd-50a6-485d-b960-ebda8e5944af") : secret "maas-api-e2e-migrate-e62fbb04-serving-cert" not found
Sep 09 21:05:30.921570 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.921541    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"]
Sep 09 21:05:30.924963 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.924946    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:30.927268 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.927246    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e62fbb04\""
Sep 09 21:05:30.927437 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.927419    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e62fbb04-dockercfg-mwwz5\""
Sep 09 21:05:30.935719 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.935691    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"]
Sep 09 21:05:30.937690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.937669    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"]
Sep 09 21:05:30.941009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.940991    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:30.951818 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.951797    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"]
Sep 09 21:05:30.965031 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.965005    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c301f8c0-9d01-4ef6-b287-2737837d2879-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:30.965247 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:30.965223    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrqn\" (UniqueName: \"kubernetes.io/projected/c301f8c0-9d01-4ef6-b287-2737837d2879-kube-api-access-mqrqn\") pod \"payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:31.066532 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.066500    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrqn\" (UniqueName: \"kubernetes.io/projected/c301f8c0-9d01-4ef6-b287-2737837d2879-kube-api-access-mqrqn\") pod \"payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:31.066720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.066546    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c301f8c0-9d01-4ef6-b287-2737837d2879-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:31.066720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.066577    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb9k\" (UniqueName: \"kubernetes.io/projected/0d64f947-d134-4d4a-84da-c1f2fe6019b6-kube-api-access-kwb9k\") pod \"payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.066909 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.066832    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64f947-d134-4d4a-84da-c1f2fe6019b6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.067407 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.067387    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c301f8c0-9d01-4ef6-b287-2737837d2879-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:31.076219 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.075998    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrqn\" (UniqueName: \"kubernetes.io/projected/c301f8c0-9d01-4ef6-b287-2737837d2879-kube-api-access-mqrqn\") pod \"payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:31.168256 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.168165    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb9k\" (UniqueName: \"kubernetes.io/projected/0d64f947-d134-4d4a-84da-c1f2fe6019b6-kube-api-access-kwb9k\") pod \"payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.168256 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.168211    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64f947-d134-4d4a-84da-c1f2fe6019b6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.168882 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.168861    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64f947-d134-4d4a-84da-c1f2fe6019b6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.176693 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.176666    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb9k\" (UniqueName: \"kubernetes.io/projected/0d64f947-d134-4d4a-84da-c1f2fe6019b6-kube-api-access-kwb9k\") pod \"payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") " pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.236670 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.236640    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:31.253969 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.253939    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:31.514827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.514801    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"]
Sep 09 21:05:31.517216 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:05:31.517190    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc301f8c0_9d01_4ef6_b287_2737837d2879.slice/crio-1f827c11a963238d13c990c9b66b71bb3028d796c40a7139618fbdedc76d3475 WatchSource:0}: Error finding container 1f827c11a963238d13c990c9b66b71bb3028d796c40a7139618fbdedc76d3475: Status 404 returned error can't find the container with id 1f827c11a963238d13c990c9b66b71bb3028d796c40a7139618fbdedc76d3475
Sep 09 21:05:31.554129 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.554102    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"]
Sep 09 21:05:31.557106 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:05:31.557074    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d64f947_d134_4d4a_84da_c1f2fe6019b6.slice/crio-080a5554c15da655f7d0a82ccb6625033354ddc67edb9b51e4200e9e5c236d3b WatchSource:0}: Error finding container 080a5554c15da655f7d0a82ccb6625033354ddc67edb9b51e4200e9e5c236d3b: Status 404 returned error can't find the container with id 080a5554c15da655f7d0a82ccb6625033354ddc67edb9b51e4200e9e5c236d3b
Sep 09 21:05:31.876494 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.876400    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:31.878953 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:31.878923    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") pod \"maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:32.020009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.019963    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:32.208296 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.208261    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc" event={"ID":"c301f8c0-9d01-4ef6-b287-2737837d2879","Type":"ContainerStarted","Data":"812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74"}
Sep 09 21:05:32.208535 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.208326    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc" event={"ID":"c301f8c0-9d01-4ef6-b287-2737837d2879","Type":"ContainerStarted","Data":"1f827c11a963238d13c990c9b66b71bb3028d796c40a7139618fbdedc76d3475"}
Sep 09 21:05:32.208535 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.208362    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:32.209988 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.209964    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b" event={"ID":"0d64f947-d134-4d4a-84da-c1f2fe6019b6","Type":"ContainerStarted","Data":"13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8"}
Sep 09 21:05:32.210121 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.210001    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b" event={"ID":"0d64f947-d134-4d4a-84da-c1f2fe6019b6","Type":"ContainerStarted","Data":"080a5554c15da655f7d0a82ccb6625033354ddc67edb9b51e4200e9e5c236d3b"}
Sep 09 21:05:32.210121 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.210091    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:05:32.224501 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.224460    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc" podStartSLOduration=2.224447325 podStartE2EDuration="2.224447325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:32.222590278 +0000 UTC m=+1020.725420280" watchObservedRunningTime="2026-09-09 21:05:32.224447325 +0000 UTC m=+1020.727277325"
Sep 09 21:05:32.248937 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.245876    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b" podStartSLOduration=2.245857725 podStartE2EDuration="2.245857725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:32.244099139 +0000 UTC m=+1020.746929140" watchObservedRunningTime="2026-09-09 21:05:32.245857725 +0000 UTC m=+1020.748687734"
Sep 09 21:05:32.309385 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:32.309357    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"]
Sep 09 21:05:32.310832 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:05:32.310803    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f97f6bd_50a6_485d_b960_ebda8e5944af.slice/crio-48c88818519ef56ef71600bf95eb1aa10b7acdd1930aeebb7bafe3c1d58fdcb6 WatchSource:0}: Error finding container 48c88818519ef56ef71600bf95eb1aa10b7acdd1930aeebb7bafe3c1d58fdcb6: Status 404 returned error can't find the container with id 48c88818519ef56ef71600bf95eb1aa10b7acdd1930aeebb7bafe3c1d58fdcb6
Sep 09 21:05:33.222646 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:33.222608    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" event={"ID":"4f97f6bd-50a6-485d-b960-ebda8e5944af","Type":"ContainerStarted","Data":"c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8"}
Sep 09 21:05:33.222820 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:33.222691    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" event={"ID":"4f97f6bd-50a6-485d-b960-ebda8e5944af","Type":"ContainerStarted","Data":"48c88818519ef56ef71600bf95eb1aa10b7acdd1930aeebb7bafe3c1d58fdcb6"}
Sep 09 21:05:33.243401 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:33.243356    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" podStartSLOduration=2.834204866 podStartE2EDuration="3.243343283s" totalImagesPullingTime="409.138417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:33.24175503 +0000 UTC m=+1021.744585028" watchObservedRunningTime="2026-09-09 21:05:33.243343283 +0000 UTC m=+1021.746173283"
Sep 09 21:05:34.228727 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:34.228691    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:40.239830 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:40.239799    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:05:43.225312 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:43.225277    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:05:43.225681 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:05:43.225361    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:06:11.555305 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.555260    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"]
Sep 09 21:06:11.555847 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.555485    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" podUID="4f97f6bd-50a6-485d-b960-ebda8e5944af" containerName="maas-api" containerID="cri-o://c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8" gracePeriod=30
Sep 09 21:06:11.614772 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.614722    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"]
Sep 09 21:06:11.615071 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.615034    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b" podUID="0d64f947-d134-4d4a-84da-c1f2fe6019b6" containerName="payload-processing" containerID="cri-o://13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8" gracePeriod=30
Sep 09 21:06:11.622869 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.622841    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"]
Sep 09 21:06:11.623091 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.623062    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc" podUID="c301f8c0-9d01-4ef6-b287-2737837d2879" containerName="payload-pre-processing" containerID="cri-o://812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74" gracePeriod=30
Sep 09 21:06:11.904826 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.904798    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:06:11.953799 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.953644    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:06:11.988554 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.988527    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:06:11.989108 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.989001    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spw6m\" (UniqueName: \"kubernetes.io/projected/4f97f6bd-50a6-485d-b960-ebda8e5944af-kube-api-access-spw6m\") pod \"4f97f6bd-50a6-485d-b960-ebda8e5944af\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") "
Sep 09 21:06:11.989108 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.989047    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-metrics-certs\") pod \"4f97f6bd-50a6-485d-b960-ebda8e5944af\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") "
Sep 09 21:06:11.989328 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.989308    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") pod \"4f97f6bd-50a6-485d-b960-ebda8e5944af\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") "
Sep 09 21:06:11.989510 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.989495    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f97f6bd-50a6-485d-b960-ebda8e5944af-openshift-service-ca\") pod \"4f97f6bd-50a6-485d-b960-ebda8e5944af\" (UID: \"4f97f6bd-50a6-485d-b960-ebda8e5944af\") "
Sep 09 21:06:11.989971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.989942    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f97f6bd-50a6-485d-b960-ebda8e5944af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f97f6bd-50a6-485d-b960-ebda8e5944af" (UID: "4f97f6bd-50a6-485d-b960-ebda8e5944af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:11.991926 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.991878    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f97f6bd-50a6-485d-b960-ebda8e5944af" (UID: "4f97f6bd-50a6-485d-b960-ebda8e5944af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:11.991926 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.991911    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f97f6bd-50a6-485d-b960-ebda8e5944af-kube-api-access-spw6m" (OuterVolumeSpecName: "kube-api-access-spw6m") pod "4f97f6bd-50a6-485d-b960-ebda8e5944af" (UID: "4f97f6bd-50a6-485d-b960-ebda8e5944af"). InnerVolumeSpecName "kube-api-access-spw6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:11.992101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:11.992052    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f97f6bd-50a6-485d-b960-ebda8e5944af" (UID: "4f97f6bd-50a6-485d-b960-ebda8e5944af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:12.090560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.090486    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64f947-d134-4d4a-84da-c1f2fe6019b6-plugins-config-volume\") pod \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") "
Sep 09 21:06:12.090704 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.090556    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c301f8c0-9d01-4ef6-b287-2737837d2879-plugins-config-volume\") pod \"c301f8c0-9d01-4ef6-b287-2737837d2879\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") "
Sep 09 21:06:12.090704 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.090599    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwb9k\" (UniqueName: \"kubernetes.io/projected/0d64f947-d134-4d4a-84da-c1f2fe6019b6-kube-api-access-kwb9k\") pod \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\" (UID: \"0d64f947-d134-4d4a-84da-c1f2fe6019b6\") "
Sep 09 21:06:12.090704 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.090651    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqrqn\" (UniqueName: \"kubernetes.io/projected/c301f8c0-9d01-4ef6-b287-2737837d2879-kube-api-access-mqrqn\") pod \"c301f8c0-9d01-4ef6-b287-2737837d2879\" (UID: \"c301f8c0-9d01-4ef6-b287-2737837d2879\") "
Sep 09 21:06:12.090955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.090916    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d64f947-d134-4d4a-84da-c1f2fe6019b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d64f947-d134-4d4a-84da-c1f2fe6019b6" (UID: "0d64f947-d134-4d4a-84da-c1f2fe6019b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:12.090955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.090944    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c301f8c0-9d01-4ef6-b287-2737837d2879-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c301f8c0-9d01-4ef6-b287-2737837d2879" (UID: "c301f8c0-9d01-4ef6-b287-2737837d2879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:12.091381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.091295    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-maas-api-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.091381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.091319    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f97f6bd-50a6-485d-b960-ebda8e5944af-openshift-service-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.091381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.091334    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d64f947-d134-4d4a-84da-c1f2fe6019b6-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.091381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.091348    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c301f8c0-9d01-4ef6-b287-2737837d2879-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.091381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.091363    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spw6m\" (UniqueName: \"kubernetes.io/projected/4f97f6bd-50a6-485d-b960-ebda8e5944af-kube-api-access-spw6m\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.091381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.091378    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f97f6bd-50a6-485d-b960-ebda8e5944af-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.092901 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.092877    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d64f947-d134-4d4a-84da-c1f2fe6019b6-kube-api-access-kwb9k" (OuterVolumeSpecName: "kube-api-access-kwb9k") pod "0d64f947-d134-4d4a-84da-c1f2fe6019b6" (UID: "0d64f947-d134-4d4a-84da-c1f2fe6019b6"). InnerVolumeSpecName "kube-api-access-kwb9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:12.092982 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.092915    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c301f8c0-9d01-4ef6-b287-2737837d2879-kube-api-access-mqrqn" (OuterVolumeSpecName: "kube-api-access-mqrqn") pod "c301f8c0-9d01-4ef6-b287-2737837d2879" (UID: "c301f8c0-9d01-4ef6-b287-2737837d2879"). InnerVolumeSpecName "kube-api-access-mqrqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:12.191866 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.191843    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb9k\" (UniqueName: \"kubernetes.io/projected/0d64f947-d134-4d4a-84da-c1f2fe6019b6-kube-api-access-kwb9k\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.191866 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.191866    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrqn\" (UniqueName: \"kubernetes.io/projected/c301f8c0-9d01-4ef6-b287-2737837d2879-kube-api-access-mqrqn\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:12.454086 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.454053    2548 generic.go:356] "Generic (PLEG): container finished" podID="0d64f947-d134-4d4a-84da-c1f2fe6019b6" containerID="13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8" exitCode=0
Sep 09 21:06:12.454263 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.454100    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"
Sep 09 21:06:12.454263 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.454135    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b" event={"ID":"0d64f947-d134-4d4a-84da-c1f2fe6019b6","Type":"ContainerDied","Data":"13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8"}
Sep 09 21:06:12.454263 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.454198    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b" event={"ID":"0d64f947-d134-4d4a-84da-c1f2fe6019b6","Type":"ContainerDied","Data":"080a5554c15da655f7d0a82ccb6625033354ddc67edb9b51e4200e9e5c236d3b"}
Sep 09 21:06:12.454263 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.454235    2548 scope.go:117] "RemoveContainer" containerID="13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8"
Sep 09 21:06:12.455971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.455946    2548 generic.go:356] "Generic (PLEG): container finished" podID="4f97f6bd-50a6-485d-b960-ebda8e5944af" containerID="c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8" exitCode=0
Sep 09 21:06:12.456103 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.455998    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"
Sep 09 21:06:12.456103 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.456039    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" event={"ID":"4f97f6bd-50a6-485d-b960-ebda8e5944af","Type":"ContainerDied","Data":"c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8"}
Sep 09 21:06:12.456219 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.456106    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff" event={"ID":"4f97f6bd-50a6-485d-b960-ebda8e5944af","Type":"ContainerDied","Data":"48c88818519ef56ef71600bf95eb1aa10b7acdd1930aeebb7bafe3c1d58fdcb6"}
Sep 09 21:06:12.457808 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.457782    2548 generic.go:356] "Generic (PLEG): container finished" podID="c301f8c0-9d01-4ef6-b287-2737837d2879" containerID="812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74" exitCode=0
Sep 09 21:06:12.457929 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.457850    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc" event={"ID":"c301f8c0-9d01-4ef6-b287-2737837d2879","Type":"ContainerDied","Data":"812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74"}
Sep 09 21:06:12.457929 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.457863    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"
Sep 09 21:06:12.457929 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.457892    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc" event={"ID":"c301f8c0-9d01-4ef6-b287-2737837d2879","Type":"ContainerDied","Data":"1f827c11a963238d13c990c9b66b71bb3028d796c40a7139618fbdedc76d3475"}
Sep 09 21:06:12.465065 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.465046    2548 scope.go:117] "RemoveContainer" containerID="13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8"
Sep 09 21:06:12.465338 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:12.465315    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8\": container with ID starting with 13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8 not found: ID does not exist" containerID="13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8"
Sep 09 21:06:12.465418 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.465345    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8"} err="failed to get container status \"13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8\": rpc error: code = NotFound desc = could not find container \"13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8\": container with ID starting with 13c38d07c6ce77ffb742f67a10bc964ff7b3ec72b30ed939c678079d5ded9fc8 not found: ID does not exist"
Sep 09 21:06:12.465418 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.465361    2548 scope.go:117] "RemoveContainer" containerID="c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8"
Sep 09 21:06:12.474724 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.474706    2548 scope.go:117] "RemoveContainer" containerID="c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8"
Sep 09 21:06:12.475003 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:12.474982    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8\": container with ID starting with c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8 not found: ID does not exist" containerID="c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8"
Sep 09 21:06:12.475079 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.475012    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8"} err="failed to get container status \"c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8\": rpc error: code = NotFound desc = could not find container \"c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8\": container with ID starting with c9f4bf17d50b6ad4f27fe97c0a6f9f3f76aa545d82dd9939cd43dbcc48d73ba8 not found: ID does not exist"
Sep 09 21:06:12.475079 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.475036    2548 scope.go:117] "RemoveContainer" containerID="812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74"
Sep 09 21:06:12.478410 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.478393    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"]
Sep 09 21:06:12.481946 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.481929    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e62fbb04-7d5c96c9d-9nn7b"]
Sep 09 21:06:12.483248 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.483232    2548 scope.go:117] "RemoveContainer" containerID="812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74"
Sep 09 21:06:12.483486 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:12.483471    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74\": container with ID starting with 812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74 not found: ID does not exist" containerID="812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74"
Sep 09 21:06:12.483535 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.483491    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74"} err="failed to get container status \"812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74\": rpc error: code = NotFound desc = could not find container \"812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74\": container with ID starting with 812a4a40f1534767aaee778447c2282fb582c5d7adf7fada804a506aa6d83e74 not found: ID does not exist"
Sep 09 21:06:12.495540 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.495520    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"]
Sep 09 21:06:12.499480 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.499457    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e62fbb04-6674cf6c79-7wgff"]
Sep 09 21:06:12.509405 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.509387    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"]
Sep 09 21:06:12.513584 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:12.513565    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e62fbb04-7bd4b896f8-bcdjc"]
Sep 09 21:06:13.969432 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:13.969400    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d64f947-d134-4d4a-84da-c1f2fe6019b6" path="/var/lib/kubelet/pods/0d64f947-d134-4d4a-84da-c1f2fe6019b6/volumes"
Sep 09 21:06:13.969874 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:13.969859    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f97f6bd-50a6-485d-b960-ebda8e5944af" path="/var/lib/kubelet/pods/4f97f6bd-50a6-485d-b960-ebda8e5944af/volumes"
Sep 09 21:06:13.970219 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:13.970203    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c301f8c0-9d01-4ef6-b287-2737837d2879" path="/var/lib/kubelet/pods/c301f8c0-9d01-4ef6-b287-2737837d2879/volumes"
Sep 09 21:06:16.414279 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414243    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"]
Sep 09 21:06:16.414759 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414710    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d64f947-d134-4d4a-84da-c1f2fe6019b6" containerName="payload-processing"
Sep 09 21:06:16.414819 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414764    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d64f947-d134-4d4a-84da-c1f2fe6019b6" containerName="payload-processing"
Sep 09 21:06:16.414819 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414809    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c301f8c0-9d01-4ef6-b287-2737837d2879" containerName="payload-pre-processing"
Sep 09 21:06:16.414819 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414818    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="c301f8c0-9d01-4ef6-b287-2737837d2879" containerName="payload-pre-processing"
Sep 09 21:06:16.414940 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414833    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f97f6bd-50a6-485d-b960-ebda8e5944af" containerName="maas-api"
Sep 09 21:06:16.414940 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414842    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f97f6bd-50a6-485d-b960-ebda8e5944af" containerName="maas-api"
Sep 09 21:06:16.415008 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414941    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="c301f8c0-9d01-4ef6-b287-2737837d2879" containerName="payload-pre-processing"
Sep 09 21:06:16.415008 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414952    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f97f6bd-50a6-485d-b960-ebda8e5944af" containerName="maas-api"
Sep 09 21:06:16.415008 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.414957    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d64f947-d134-4d4a-84da-c1f2fe6019b6" containerName="payload-processing"
Sep 09 21:06:16.420470 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.420447    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.423045 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.423014    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-857dd97d-serving-cert\""
Sep 09 21:06:16.431566 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.431543    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"]
Sep 09 21:06:16.533070 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.533037    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.533229 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.533088    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-metrics-certs\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.533229 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.533175    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-openshift-service-ca\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.533316 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.533255    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jsc\" (UniqueName: \"kubernetes.io/projected/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-kube-api-access-d2jsc\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.634384 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.634349    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.634556 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.634412    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-metrics-certs\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.634556 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.634457    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-openshift-service-ca\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.634556 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:16.634490    2548 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-serving-cert: secret "maas-api-e2e-ait-857dd97d-serving-cert" not found
Sep 09 21:06:16.634556 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.634528    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jsc\" (UniqueName: \"kubernetes.io/projected/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-kube-api-access-d2jsc\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.634775 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:16.634562    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls podName:cb9822a7-8d5b-4ead-b9de-aa2512f6cf35 nodeName:}" failed. No retries permitted until 2026-09-09 21:06:17.134541882 +0000 UTC m=+1065.637371878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls") pod "maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" (UID: "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35") : secret "maas-api-e2e-ait-857dd97d-serving-cert" not found
Sep 09 21:06:16.635284 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.635258    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-openshift-service-ca\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.636812 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.636786    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-metrics-certs\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:16.646529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:16.646502    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jsc\" (UniqueName: \"kubernetes.io/projected/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-kube-api-access-d2jsc\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:17.140347 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.140314    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:17.142666 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.142633    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls\") pod \"maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:17.210515 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.210488    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"]
Sep 09 21:06:17.214650 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.214627    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.217179 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.217156    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-80db9b\"/\"default-dockercfg-c7v58\""
Sep 09 21:06:17.217762 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.217718    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-80db9b\"/\"autores-model-80db9b-kserve-self-signed-certs\""
Sep 09 21:06:17.217878 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.217718    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-80db9b\"/\"kube-root-ca.crt\""
Sep 09 21:06:17.218577 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.218560    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-80db9b\"/\"openshift-service-ca.crt\""
Sep 09 21:06:17.226627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.226609    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"]
Sep 09 21:06:17.332465 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.332433    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:17.332614 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.332575    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"]
Sep 09 21:06:17.335998 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.335980    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.341872 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.341845    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-home\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.341992 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.341892    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-tmp-dir\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.342059 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.341988    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74dfcc6-154a-465e-b681-aef1e3f7e622-tls-certs\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.342139 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.342118    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt868\" (UniqueName: \"kubernetes.io/projected/f74dfcc6-154a-465e-b681-aef1e3f7e622-kube-api-access-xt868\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.342197 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.342162    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-dshm\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.342197 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.342184    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-model-cache\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.342343 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.342328    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-857dd97d-dockercfg-sq68p\""
Sep 09 21:06:17.342410 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.342364    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-857dd97d\""
Sep 09 21:06:17.350773 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.350749    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"]
Sep 09 21:06:17.443602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443563    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74dfcc6-154a-465e-b681-aef1e3f7e622-tls-certs\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443724    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb52238-b4ee-4cec-ba95-03dd023b6a95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443794    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt868\" (UniqueName: \"kubernetes.io/projected/f74dfcc6-154a-465e-b681-aef1e3f7e622-kube-api-access-xt868\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443832    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-dshm\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443859    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-model-cache\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443895    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-home\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443963    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thrh\" (UniqueName: \"kubernetes.io/projected/bbb52238-b4ee-4cec-ba95-03dd023b6a95-kube-api-access-5thrh\") pod \"payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.444034 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.443989    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-tmp-dir\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.444176    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-model-cache\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.444311    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-tmp-dir\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.444412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.444372    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-home\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.448159 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.448115    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74dfcc6-154a-465e-b681-aef1e3f7e622-tls-certs\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.448478 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.448455    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-dshm\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.453855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.453776    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt868\" (UniqueName: \"kubernetes.io/projected/f74dfcc6-154a-465e-b681-aef1e3f7e622-kube-api-access-xt868\") pod \"autores-model-80db9b-kserve-779649bb75-lbzpp\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") " pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.527720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.527686    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:17.545412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.545379    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thrh\" (UniqueName: \"kubernetes.io/projected/bbb52238-b4ee-4cec-ba95-03dd023b6a95-kube-api-access-5thrh\") pod \"payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.546437 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.545529    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb52238-b4ee-4cec-ba95-03dd023b6a95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.546437 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.546390    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb52238-b4ee-4cec-ba95-03dd023b6a95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.554370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.554350    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thrh\" (UniqueName: \"kubernetes.io/projected/bbb52238-b4ee-4cec-ba95-03dd023b6a95-kube-api-access-5thrh\") pod \"payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.634158 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.634130    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"]
Sep 09 21:06:17.636709 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:06:17.636669    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9822a7_8d5b_4ead_b9de_aa2512f6cf35.slice/crio-7a7b3f86f4e11c8d627fb7d98b5bc5ac0194cd5f422e9d7713b3e1a177640999 WatchSource:0}: Error finding container 7a7b3f86f4e11c8d627fb7d98b5bc5ac0194cd5f422e9d7713b3e1a177640999: Status 404 returned error can't find the container with id 7a7b3f86f4e11c8d627fb7d98b5bc5ac0194cd5f422e9d7713b3e1a177640999
Sep 09 21:06:17.674975 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.674912    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:17.813119 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.813092    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"]
Sep 09 21:06:17.815206 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:06:17.815178    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf74dfcc6_154a_465e_b681_aef1e3f7e622.slice/crio-44b7d40f1bffdca10bb2f32afd737e63d1215f64a4cd0cadbcbdde5114438e15 WatchSource:0}: Error finding container 44b7d40f1bffdca10bb2f32afd737e63d1215f64a4cd0cadbcbdde5114438e15: Status 404 returned error can't find the container with id 44b7d40f1bffdca10bb2f32afd737e63d1215f64a4cd0cadbcbdde5114438e15
Sep 09 21:06:17.952913 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:17.952882    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"]
Sep 09 21:06:17.955319 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:06:17.955292    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb52238_b4ee_4cec_ba95_03dd023b6a95.slice/crio-872fb32f8145362625e96f198973a629ecfd648065fe477350332436ec55354f WatchSource:0}: Error finding container 872fb32f8145362625e96f198973a629ecfd648065fe477350332436ec55354f: Status 404 returned error can't find the container with id 872fb32f8145362625e96f198973a629ecfd648065fe477350332436ec55354f
Sep 09 21:06:18.494849 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.494808    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl" event={"ID":"bbb52238-b4ee-4cec-ba95-03dd023b6a95","Type":"ContainerStarted","Data":"3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a"}
Sep 09 21:06:18.495276 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.494882    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:18.495276 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.494905    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl" event={"ID":"bbb52238-b4ee-4cec-ba95-03dd023b6a95","Type":"ContainerStarted","Data":"872fb32f8145362625e96f198973a629ecfd648065fe477350332436ec55354f"}
Sep 09 21:06:18.496351 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.496323    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" event={"ID":"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35","Type":"ContainerStarted","Data":"f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8"}
Sep 09 21:06:18.496470 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.496371    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" event={"ID":"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35","Type":"ContainerStarted","Data":"7a7b3f86f4e11c8d627fb7d98b5bc5ac0194cd5f422e9d7713b3e1a177640999"}
Sep 09 21:06:18.496470 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.496430    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:18.497755 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.497706    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp" event={"ID":"f74dfcc6-154a-465e-b681-aef1e3f7e622","Type":"ContainerStarted","Data":"62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f"}
Sep 09 21:06:18.497841 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.497768    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp" event={"ID":"f74dfcc6-154a-465e-b681-aef1e3f7e622","Type":"ContainerStarted","Data":"44b7d40f1bffdca10bb2f32afd737e63d1215f64a4cd0cadbcbdde5114438e15"}
Sep 09 21:06:18.513576 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.513536    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl" podStartSLOduration=1.513522865 podStartE2EDuration="1.513522865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:18.51247617 +0000 UTC m=+1067.015306171" watchObservedRunningTime="2026-09-09 21:06:18.513522865 +0000 UTC m=+1067.016352863"
Sep 09 21:06:18.531029 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.530985    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" podStartSLOduration=2.1687731980000002 podStartE2EDuration="2.530971489s" totalImagesPullingTime="362.198291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:18.529003023 +0000 UTC m=+1067.031833025" watchObservedRunningTime="2026-09-09 21:06:18.530971489 +0000 UTC m=+1067.033801487"
Sep 09 21:06:18.547191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:18.547151    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp" podStartSLOduration=1.54713861 podStartE2EDuration="1.54713861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:18.545849743 +0000 UTC m=+1067.048679746" watchObservedRunningTime="2026-09-09 21:06:18.54713861 +0000 UTC m=+1067.049968610"
Sep 09 21:06:24.508212 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:24.508180    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:27.528295 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:27.528257    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:27.528666 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:27.528309    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:27.540536 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:27.540513    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:27.560753 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:27.560709    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:06:29.505298 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:29.505269    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:47.183785 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.183747    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"]
Sep 09 21:06:47.184333 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.183960    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" podUID="cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" containerName="maas-api" containerID="cri-o://f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8" gracePeriod=30
Sep 09 21:06:47.280527 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.280489    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"]
Sep 09 21:06:47.281028 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.280778    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl" podUID="bbb52238-b4ee-4cec-ba95-03dd023b6a95" containerName="payload-pre-processing" containerID="cri-o://3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a" gracePeriod=30
Sep 09 21:06:47.460859 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.460727    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:47.538950 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.538910    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls\") pod \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") "
Sep 09 21:06:47.539104 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.539020    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-openshift-service-ca\") pod \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") "
Sep 09 21:06:47.539104 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.539074    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-metrics-certs\") pod \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") "
Sep 09 21:06:47.539217 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.539156    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2jsc\" (UniqueName: \"kubernetes.io/projected/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-kube-api-access-d2jsc\") pod \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\" (UID: \"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35\") "
Sep 09 21:06:47.539627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.539577    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" (UID: "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:47.541293 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.541270    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" (UID: "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:47.541483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.541465    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:47.541603 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.541566    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" (UID: "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:47.541703 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.541602    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-kube-api-access-d2jsc" (OuterVolumeSpecName: "kube-api-access-d2jsc") pod "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" (UID: "cb9822a7-8d5b-4ead-b9de-aa2512f6cf35"). InnerVolumeSpecName "kube-api-access-d2jsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:47.639856 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.639815    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb52238-b4ee-4cec-ba95-03dd023b6a95-plugins-config-volume\") pod \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") "
Sep 09 21:06:47.639856 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.639859    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5thrh\" (UniqueName: \"kubernetes.io/projected/bbb52238-b4ee-4cec-ba95-03dd023b6a95-kube-api-access-5thrh\") pod \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\" (UID: \"bbb52238-b4ee-4cec-ba95-03dd023b6a95\") "
Sep 09 21:06:47.640285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.640259    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb52238-b4ee-4cec-ba95-03dd023b6a95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbb52238-b4ee-4cec-ba95-03dd023b6a95" (UID: "bbb52238-b4ee-4cec-ba95-03dd023b6a95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:06:47.640285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.640281    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:47.640397 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.640300    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2jsc\" (UniqueName: \"kubernetes.io/projected/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-kube-api-access-d2jsc\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:47.640397 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.640311    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-maas-api-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:47.640397 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.640321    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35-openshift-service-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:47.641961 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.641930    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb52238-b4ee-4cec-ba95-03dd023b6a95-kube-api-access-5thrh" (OuterVolumeSpecName: "kube-api-access-5thrh") pod "bbb52238-b4ee-4cec-ba95-03dd023b6a95" (UID: "bbb52238-b4ee-4cec-ba95-03dd023b6a95"). InnerVolumeSpecName "kube-api-access-5thrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:47.667028 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.666993    2548 generic.go:356] "Generic (PLEG): container finished" podID="bbb52238-b4ee-4cec-ba95-03dd023b6a95" containerID="3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a" exitCode=0
Sep 09 21:06:47.667174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.667044    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"
Sep 09 21:06:47.667174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.667073    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl" event={"ID":"bbb52238-b4ee-4cec-ba95-03dd023b6a95","Type":"ContainerDied","Data":"3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a"}
Sep 09 21:06:47.667174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.667131    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl" event={"ID":"bbb52238-b4ee-4cec-ba95-03dd023b6a95","Type":"ContainerDied","Data":"872fb32f8145362625e96f198973a629ecfd648065fe477350332436ec55354f"}
Sep 09 21:06:47.667174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.667163    2548 scope.go:117] "RemoveContainer" containerID="3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a"
Sep 09 21:06:47.668626 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.668594    2548 generic.go:356] "Generic (PLEG): container finished" podID="cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" containerID="f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8" exitCode=0
Sep 09 21:06:47.668720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.668632    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" event={"ID":"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35","Type":"ContainerDied","Data":"f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8"}
Sep 09 21:06:47.668720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.668673    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f" event={"ID":"cb9822a7-8d5b-4ead-b9de-aa2512f6cf35","Type":"ContainerDied","Data":"7a7b3f86f4e11c8d627fb7d98b5bc5ac0194cd5f422e9d7713b3e1a177640999"}
Sep 09 21:06:47.668720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.668696    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"
Sep 09 21:06:47.676556 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.676455    2548 scope.go:117] "RemoveContainer" containerID="3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a"
Sep 09 21:06:47.676871 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:47.676846    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a\": container with ID starting with 3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a not found: ID does not exist" containerID="3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a"
Sep 09 21:06:47.676973 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.676885    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a"} err="failed to get container status \"3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a\": rpc error: code = NotFound desc = could not find container \"3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a\": container with ID starting with 3549c74969e4fc1291ca7d9e855acdb97563985d68fe093f34e22ff71f60875a not found: ID does not exist"
Sep 09 21:06:47.676973 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.676901    2548 scope.go:117] "RemoveContainer" containerID="f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8"
Sep 09 21:06:47.685906 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.685884    2548 scope.go:117] "RemoveContainer" containerID="f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8"
Sep 09 21:06:47.686174 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:06:47.686152    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8\": container with ID starting with f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8 not found: ID does not exist" containerID="f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8"
Sep 09 21:06:47.686264 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.686180    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8"} err="failed to get container status \"f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8\": rpc error: code = NotFound desc = could not find container \"f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8\": container with ID starting with f5c9f977e829d200fa749b17870041f3ca8f6f24e770f5466d416e69324816f8 not found: ID does not exist"
Sep 09 21:06:47.693472 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.693437    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"]
Sep 09 21:06:47.698523 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.698491    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-857dd97d-55cf5d4fbf-fw27f"]
Sep 09 21:06:47.707290 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.707261    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"]
Sep 09 21:06:47.709001 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.708978    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-857dd97d-5fdb6dfb6-m5mgl"]
Sep 09 21:06:47.740908 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.740874    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb52238-b4ee-4cec-ba95-03dd023b6a95-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:47.740908 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.740900    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thrh\" (UniqueName: \"kubernetes.io/projected/bbb52238-b4ee-4cec-ba95-03dd023b6a95-kube-api-access-5thrh\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:06:47.968478 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.968381    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb52238-b4ee-4cec-ba95-03dd023b6a95" path="/var/lib/kubelet/pods/bbb52238-b4ee-4cec-ba95-03dd023b6a95/volumes"
Sep 09 21:06:47.968930 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:47.968909    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" path="/var/lib/kubelet/pods/cb9822a7-8d5b-4ead-b9de-aa2512f6cf35/volumes"
Sep 09 21:06:49.290025 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.289992    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"]
Sep 09 21:06:49.290499 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.290484    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb52238-b4ee-4cec-ba95-03dd023b6a95" containerName="payload-pre-processing"
Sep 09 21:06:49.290557 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.290501    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb52238-b4ee-4cec-ba95-03dd023b6a95" containerName="payload-pre-processing"
Sep 09 21:06:49.290557 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.290519    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" containerName="maas-api"
Sep 09 21:06:49.290557 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.290525    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" containerName="maas-api"
Sep 09 21:06:49.290660 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.290588    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb9822a7-8d5b-4ead-b9de-aa2512f6cf35" containerName="maas-api"
Sep 09 21:06:49.290660 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.290600    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb52238-b4ee-4cec-ba95-03dd023b6a95" containerName="payload-pre-processing"
Sep 09 21:06:49.295526 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.295502    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.298143 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.298122    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-47d82e7b-openshift-default-dockercfg-hd8c8\""
Sep 09 21:06:49.310596 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.310573    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"]
Sep 09 21:06:49.357695 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.357663    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-envoy\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.357870 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.357722    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqjnz\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-kube-api-access-kqjnz\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.357870 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.357778    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-token\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.357870 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.357851    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istiod-ca-cert\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.358044 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.357944    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-certs\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.358044 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.357987    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-podinfo\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.358044 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.358020    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-credential-socket\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.358191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.358048    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-socket\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.358191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.358098    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-data\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459250    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqjnz\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-kube-api-access-kqjnz\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459291    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-token\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459330    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istiod-ca-cert\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459374    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-certs\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459391    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-podinfo\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459417    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-credential-socket\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459437    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-socket\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.459538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459481    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-data\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.460380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459547    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-envoy\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.460380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.459871    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-credential-socket\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.460380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.460008    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-socket\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.460380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.460020    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-data\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.460380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.460075    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-certs\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.460380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.460306    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istiod-ca-cert\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.461843 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.461814    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-envoy\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.461997 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.461977    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-podinfo\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.468121 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.468100    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-token\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.468222 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.468158    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqjnz\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-kube-api-access-kqjnz\") pod \"e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") " pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.605989 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.605913    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:49.888278 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.888246    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"]
Sep 09 21:06:49.892347 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:06:49.892281    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98bc48c_b65f_4c4d_890b_dc68b843ad15.slice/crio-363ad1e69c92f8b70cd6161598593a8448571b24617e163d619541c5cfeb12b9 WatchSource:0}: Error finding container 363ad1e69c92f8b70cd6161598593a8448571b24617e163d619541c5cfeb12b9: Status 404 returned error can't find the container with id 363ad1e69c92f8b70cd6161598593a8448571b24617e163d619541c5cfeb12b9
Sep 09 21:06:49.894895 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.894857    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:06:49.895018 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.894952    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:06:49.895018 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:49.894998    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:06:50.689479 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:50.689441    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" event={"ID":"b98bc48c-b65f-4c4d-890b-dc68b843ad15","Type":"ContainerStarted","Data":"8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"}
Sep 09 21:06:50.689867 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:50.689502    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" event={"ID":"b98bc48c-b65f-4c4d-890b-dc68b843ad15","Type":"ContainerStarted","Data":"363ad1e69c92f8b70cd6161598593a8448571b24617e163d619541c5cfeb12b9"}
Sep 09 21:06:50.735246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:50.735181    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podStartSLOduration=1.735159361 podStartE2EDuration="1.735159361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:50.734621854 +0000 UTC m=+1099.237451880" watchObservedRunningTime="2026-09-09 21:06:50.735159361 +0000 UTC m=+1099.237989366"
Sep 09 21:06:51.606687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:51.606489    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:51.635977 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:51.635945    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:51.695137 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:51.695095    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:51.696294 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:51.696268    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:06:51.994623 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:51.994583    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"]
Sep 09 21:06:54.361957 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.361921    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"]
Sep 09 21:06:54.365935 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.365917    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.368589 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.368567    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-47d82e7b-dockercfg-wn5n2\""
Sep 09 21:06:54.368953 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.368939    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-47d82e7b\""
Sep 09 21:06:54.396972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.396946    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"]
Sep 09 21:06:54.513431 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.513394    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.513614 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.513464    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72slm\" (UniqueName: \"kubernetes.io/projected/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-kube-api-access-72slm\") pod \"payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.614995 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.614547    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72slm\" (UniqueName: \"kubernetes.io/projected/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-kube-api-access-72slm\") pod \"payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.615146 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.615072    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.615691 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.615673    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.625451 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.625419    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72slm\" (UniqueName: \"kubernetes.io/projected/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-kube-api-access-72slm\") pod \"payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.676411 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.676370    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:54.949656 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:54.949616    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"]
Sep 09 21:06:54.952577 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:06:54.952549    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac51e95_e4e8_4ce2_8099_0f4a4c1ae863.slice/crio-0ba7c85436920c49b8980a674ad4ab797983a54869fd5008098166da2540b870 WatchSource:0}: Error finding container 0ba7c85436920c49b8980a674ad4ab797983a54869fd5008098166da2540b870: Status 404 returned error can't find the container with id 0ba7c85436920c49b8980a674ad4ab797983a54869fd5008098166da2540b870
Sep 09 21:06:55.721626 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:55.721588    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb" event={"ID":"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863","Type":"ContainerStarted","Data":"0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9"}
Sep 09 21:06:55.722050 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:55.721660    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:06:55.722050 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:55.721685    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb" event={"ID":"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863","Type":"ContainerStarted","Data":"0ba7c85436920c49b8980a674ad4ab797983a54869fd5008098166da2540b870"}
Sep 09 21:06:55.737681 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:06:55.737625    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb" podStartSLOduration=1.737605228 podStartE2EDuration="1.737605228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:55.737330261 +0000 UTC m=+1104.240160273" watchObservedRunningTime="2026-09-09 21:06:55.737605228 +0000 UTC m=+1104.240435229"
Sep 09 21:07:06.729263 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:06.729220    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:07:07.016715 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.016599    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp" podUID="f74dfcc6-154a-465e-b681-aef1e3f7e622" containerName="main" containerID="cri-o://62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f" gracePeriod=45
Sep 09 21:07:07.327988 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.327963    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:07:07.478964 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.478917    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt868\" (UniqueName: \"kubernetes.io/projected/f74dfcc6-154a-465e-b681-aef1e3f7e622-kube-api-access-xt868\") pod \"f74dfcc6-154a-465e-b681-aef1e3f7e622\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") "
Sep 09 21:07:07.479164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479010    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-tmp-dir\") pod \"f74dfcc6-154a-465e-b681-aef1e3f7e622\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") "
Sep 09 21:07:07.479164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479077    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-dshm\") pod \"f74dfcc6-154a-465e-b681-aef1e3f7e622\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") "
Sep 09 21:07:07.479164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479125    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-home\") pod \"f74dfcc6-154a-465e-b681-aef1e3f7e622\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") "
Sep 09 21:07:07.479164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479158    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74dfcc6-154a-465e-b681-aef1e3f7e622-tls-certs\") pod \"f74dfcc6-154a-465e-b681-aef1e3f7e622\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") "
Sep 09 21:07:07.479371 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479193    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-model-cache\") pod \"f74dfcc6-154a-465e-b681-aef1e3f7e622\" (UID: \"f74dfcc6-154a-465e-b681-aef1e3f7e622\") "
Sep 09 21:07:07.479371 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479223    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f74dfcc6-154a-465e-b681-aef1e3f7e622" (UID: "f74dfcc6-154a-465e-b681-aef1e3f7e622"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:07.479371 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479349    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-home" (OuterVolumeSpecName: "home") pod "f74dfcc6-154a-465e-b681-aef1e3f7e622" (UID: "f74dfcc6-154a-465e-b681-aef1e3f7e622"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:07.479502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479470    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-model-cache" (OuterVolumeSpecName: "model-cache") pod "f74dfcc6-154a-465e-b681-aef1e3f7e622" (UID: "f74dfcc6-154a-465e-b681-aef1e3f7e622"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:07.479598 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479579    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-tmp-dir\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:07.479641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479601    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-home\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:07.479641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.479612    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-model-cache\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:07.481361 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.481328    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74dfcc6-154a-465e-b681-aef1e3f7e622-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f74dfcc6-154a-465e-b681-aef1e3f7e622" (UID: "f74dfcc6-154a-465e-b681-aef1e3f7e622"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:07.481602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.481574    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74dfcc6-154a-465e-b681-aef1e3f7e622-kube-api-access-xt868" (OuterVolumeSpecName: "kube-api-access-xt868") pod "f74dfcc6-154a-465e-b681-aef1e3f7e622" (UID: "f74dfcc6-154a-465e-b681-aef1e3f7e622"). InnerVolumeSpecName "kube-api-access-xt868". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:07.481602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.481575    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-dshm" (OuterVolumeSpecName: "dshm") pod "f74dfcc6-154a-465e-b681-aef1e3f7e622" (UID: "f74dfcc6-154a-465e-b681-aef1e3f7e622"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:07.580376 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.580287    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt868\" (UniqueName: \"kubernetes.io/projected/f74dfcc6-154a-465e-b681-aef1e3f7e622-kube-api-access-xt868\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:07.580376 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.580318    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74dfcc6-154a-465e-b681-aef1e3f7e622-dshm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:07.580376 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.580331    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74dfcc6-154a-465e-b681-aef1e3f7e622-tls-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:07.790290 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.790254    2548 generic.go:356] "Generic (PLEG): container finished" podID="f74dfcc6-154a-465e-b681-aef1e3f7e622" containerID="62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f" exitCode=0
Sep 09 21:07:07.790665 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.790310    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"
Sep 09 21:07:07.790665 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.790313    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp" event={"ID":"f74dfcc6-154a-465e-b681-aef1e3f7e622","Type":"ContainerDied","Data":"62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f"}
Sep 09 21:07:07.790665 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.790417    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp" event={"ID":"f74dfcc6-154a-465e-b681-aef1e3f7e622","Type":"ContainerDied","Data":"44b7d40f1bffdca10bb2f32afd737e63d1215f64a4cd0cadbcbdde5114438e15"}
Sep 09 21:07:07.790665 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.790434    2548 scope.go:117] "RemoveContainer" containerID="62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f"
Sep 09 21:07:07.800838 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.800820    2548 scope.go:117] "RemoveContainer" containerID="62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f"
Sep 09 21:07:07.801112 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:07:07.801093    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f\": container with ID starting with 62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f not found: ID does not exist" containerID="62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f"
Sep 09 21:07:07.801160 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.801122    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f"} err="failed to get container status \"62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f\": rpc error: code = NotFound desc = could not find container \"62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f\": container with ID starting with 62d33e532389cbe71823ede0c243f4715af9f1437b519bcc39fe647c0a45fa3f not found: ID does not exist"
Sep 09 21:07:07.811862 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.811834    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"]
Sep 09 21:07:07.815719 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.815688    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-80db9b/autores-model-80db9b-kserve-779649bb75-lbzpp"]
Sep 09 21:07:07.973766 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:07.973713    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74dfcc6-154a-465e-b681-aef1e3f7e622" path="/var/lib/kubelet/pods/f74dfcc6-154a-465e-b681-aef1e3f7e622/volumes"
Sep 09 21:07:14.839920 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:14.839882    2548 generic.go:356] "Generic (PLEG): container finished" podID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerID="8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99" exitCode=0
Sep 09 21:07:14.840352 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:14.839976    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" event={"ID":"b98bc48c-b65f-4c4d-890b-dc68b843ad15","Type":"ContainerDied","Data":"8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"}
Sep 09 21:07:14.840492 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:14.840472    2548 scope.go:117] "RemoveContainer" containerID="8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"
Sep 09 21:07:14.841673 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:14.841647    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:14.841798 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:14.841728    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:14.841798 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:14.841793    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:15.849170 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:15.849119    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" event={"ID":"b98bc48c-b65f-4c4d-890b-dc68b843ad15","Type":"ContainerStarted","Data":"aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8"}
Sep 09 21:07:16.606967 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:16.606928    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:07:16.607147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:16.607120    2548 patch_prober.go:28] interesting pod/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:07:16.607198 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:16.607173    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 21:07:17.606996 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:17.606961    2548 patch_prober.go:28] interesting pod/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:07:17.607395 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:17.607038    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 21:07:18.607007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:18.606971    2548 patch_prober.go:28] interesting pod/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:07:18.607527 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:18.607024    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 21:07:19.607036 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:19.606996    2548 patch_prober.go:28] interesting pod/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:07:19.607414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:19.607060    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 21:07:20.607270 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:20.607232    2548 patch_prober.go:28] interesting pod/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:07:20.607635 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:20.607294    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 21:07:21.606432 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:21.606391    2548 patch_prober.go:28] interesting pod/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:07:21.606615 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:21.606449    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 09 21:07:21.695661 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:21.695615    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:07:22.611325 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:22.611293    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:07:22.612222 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:22.612203    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:07:23.977533 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:23.977493    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"]
Sep 09 21:07:23.978032 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:23.977754    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb" podUID="8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" containerName="payload-pre-processing" containerID="cri-o://0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9" gracePeriod=30
Sep 09 21:07:24.235555 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.235498    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:07:24.330766 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.327799    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"]
Sep 09 21:07:24.330766 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.328126    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy" containerID="cri-o://aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8" gracePeriod=30
Sep 09 21:07:24.355026 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.354994    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-plugins-config-volume\") pod \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") "
Sep 09 21:07:24.355176 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.355068    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72slm\" (UniqueName: \"kubernetes.io/projected/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-kube-api-access-72slm\") pod \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\" (UID: \"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863\") "
Sep 09 21:07:24.355429 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.355401    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" (UID: "8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:24.357282 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.357254    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-kube-api-access-72slm" (OuterVolumeSpecName: "kube-api-access-72slm") pod "8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" (UID: "8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863"). InnerVolumeSpecName "kube-api-access-72slm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:24.456435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.456397    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.456435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.456432    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72slm\" (UniqueName: \"kubernetes.io/projected/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863-kube-api-access-72slm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:24.909725 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.909689    2548 generic.go:356] "Generic (PLEG): container finished" podID="8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" containerID="0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9" exitCode=0
Sep 09 21:07:24.909897 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.909767    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"
Sep 09 21:07:24.909897 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.909776    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb" event={"ID":"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863","Type":"ContainerDied","Data":"0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9"}
Sep 09 21:07:24.909897 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.909826    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb" event={"ID":"8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863","Type":"ContainerDied","Data":"0ba7c85436920c49b8980a674ad4ab797983a54869fd5008098166da2540b870"}
Sep 09 21:07:24.909897 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.909847    2548 scope.go:117] "RemoveContainer" containerID="0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9"
Sep 09 21:07:24.918620 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.918607    2548 scope.go:117] "RemoveContainer" containerID="0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9"
Sep 09 21:07:24.918890 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:07:24.918870    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9\": container with ID starting with 0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9 not found: ID does not exist" containerID="0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9"
Sep 09 21:07:24.918936 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.918898    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9"} err="failed to get container status \"0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9\": rpc error: code = NotFound desc = could not find container \"0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9\": container with ID starting with 0c99cd82f5d59c8fea6662136a0593a16d5104831b09145dc4e2dcf4d8bc04f9 not found: ID does not exist"
Sep 09 21:07:24.933803 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.933779    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"]
Sep 09 21:07:24.938621 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:24.938601    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-47d82e7b-5bb9469d94-66klb"]
Sep 09 21:07:25.968448 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:25.968417    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" path="/var/lib/kubelet/pods/8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863/volumes"
Sep 09 21:07:29.591241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.591217    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:07:29.710593 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710560    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-socket\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.710797 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710625    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-envoy\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.710797 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710653    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-podinfo\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.710797 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710690    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqjnz\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-kube-api-access-kqjnz\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.710797 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710762    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-token\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.711023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710811    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-data\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.711023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710829    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istiod-ca-cert\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.711023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710849    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-credential-socket\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.711023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710871    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-certs\") pod \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\" (UID: \"b98bc48c-b65f-4c4d-890b-dc68b843ad15\") "
Sep 09 21:07:29.711023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.710914    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:29.711295 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.711146    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-data" (OuterVolumeSpecName: "istio-data") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:29.711295 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.711275    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:29.711443 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.711319    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.711443 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.711338    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.711443 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.711434    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:29.711616 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.711550    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:29.712933 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.712902    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:07:29.713057 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.713036    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:29.713202 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.713182    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-kube-api-access-kqjnz" (OuterVolumeSpecName: "kube-api-access-kqjnz") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "kube-api-access-kqjnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:29.713249 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.713209    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-token" (OuterVolumeSpecName: "istio-token") pod "b98bc48c-b65f-4c4d-890b-dc68b843ad15" (UID: "b98bc48c-b65f-4c4d-890b-dc68b843ad15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:29.812413 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812380    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.812413 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812409    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.812619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812441    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqjnz\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-kube-api-access-kqjnz\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.812619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812450    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.812619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812459    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98bc48c-b65f-4c4d-890b-dc68b843ad15-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.812619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812468    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.812619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.812475    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98bc48c-b65f-4c4d-890b-dc68b843ad15-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:07:29.941507 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.941468    2548 generic.go:356] "Generic (PLEG): container finished" podID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerID="aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8" exitCode=0
Sep 09 21:07:29.941690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.941529    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"
Sep 09 21:07:29.941690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.941541    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" event={"ID":"b98bc48c-b65f-4c4d-890b-dc68b843ad15","Type":"ContainerDied","Data":"aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8"}
Sep 09 21:07:29.941690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.941626    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp" event={"ID":"b98bc48c-b65f-4c4d-890b-dc68b843ad15","Type":"ContainerDied","Data":"363ad1e69c92f8b70cd6161598593a8448571b24617e163d619541c5cfeb12b9"}
Sep 09 21:07:29.941690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.941649    2548 scope.go:117] "RemoveContainer" containerID="aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8"
Sep 09 21:07:29.953062 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.953044    2548 scope.go:117] "RemoveContainer" containerID="8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"
Sep 09 21:07:29.961488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.961470    2548 scope.go:117] "RemoveContainer" containerID="aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8"
Sep 09 21:07:29.961837 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:07:29.961785    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8\": container with ID starting with aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8 not found: ID does not exist" containerID="aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8"
Sep 09 21:07:29.961930 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.961829    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8"} err="failed to get container status \"aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8\": rpc error: code = NotFound desc = could not find container \"aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8\": container with ID starting with aeac499822803431c71e384e5a78510a9fa9988aadf3b1ffd9d17227811c53a8 not found: ID does not exist"
Sep 09 21:07:29.961930 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.961853    2548 scope.go:117] "RemoveContainer" containerID="8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"
Sep 09 21:07:29.962162 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:07:29.962143    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99\": container with ID starting with 8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99 not found: ID does not exist" containerID="8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"
Sep 09 21:07:29.962232 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.962172    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99"} err="failed to get container status \"8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99\": rpc error: code = NotFound desc = could not find container \"8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99\": container with ID starting with 8fbb8df46d5b5a04e498e01a743464d792088e86e2681d347e56ed214d611d99 not found: ID does not exist"
Sep 09 21:07:29.969405 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.969388    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"]
Sep 09 21:07:29.973485 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:29.973465    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-47d82e7b-openshift-default-6b497dfdd-48tmp"]
Sep 09 21:07:31.967861 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:07:31.967828    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" path="/var/lib/kubelet/pods/b98bc48c-b65f-4c4d-890b-dc68b843ad15/volumes"
Sep 09 21:08:02.101753 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.101653    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"]
Sep 09 21:08:02.102426 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102403    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74dfcc6-154a-465e-b681-aef1e3f7e622" containerName="main"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102431    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74dfcc6-154a-465e-b681-aef1e3f7e622" containerName="main"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102449    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102456    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102469    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102477    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102488    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" containerName="payload-pre-processing"
Sep 09 21:08:02.102529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102496    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" containerName="payload-pre-processing"
Sep 09 21:08:02.102914 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102654    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy"
Sep 09 21:08:02.102914 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102672    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74dfcc6-154a-465e-b681-aef1e3f7e622" containerName="main"
Sep 09 21:08:02.102914 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.102682    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac51e95-e4e8-4ce2-8099-0f4a4c1ae863" containerName="payload-pre-processing"
Sep 09 21:08:02.105801 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.105779    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.107916 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.107870    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0066f8\"/\"kube-root-ca.crt\""
Sep 09 21:08:02.108046 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.107998    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0066f8\"/\"openshift-service-ca.crt\""
Sep 09 21:08:02.108662 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.108640    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0066f8\"/\"explicit-model-0066f8-kserve-self-signed-certs\""
Sep 09 21:08:02.108799 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.108667    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0066f8\"/\"default-dockercfg-psx7j\""
Sep 09 21:08:02.116429 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.116397    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"]
Sep 09 21:08:02.229492 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.229464    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00311ce8-747b-4380-b6cc-d657ad987d74-tls-certs\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.229674 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.229510    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64ck\" (UniqueName: \"kubernetes.io/projected/00311ce8-747b-4380-b6cc-d657ad987d74-kube-api-access-h64ck\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.229674 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.229535    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-dshm\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.229674 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.229566    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-home\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.229819 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.229765    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-tmp-dir\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.229819 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.229801    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-model-cache\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331054 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331014    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-tmp-dir\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331054 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331055    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-model-cache\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331290 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331183    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00311ce8-747b-4380-b6cc-d657ad987d74-tls-certs\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331290 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331236    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64ck\" (UniqueName: \"kubernetes.io/projected/00311ce8-747b-4380-b6cc-d657ad987d74-kube-api-access-h64ck\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331290 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331256    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-dshm\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331299    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-home\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331430    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-model-cache\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331554 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331502    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-tmp-dir\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.331701 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.331679    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-home\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.333551 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.333528    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-dshm\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.333830 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.333815    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00311ce8-747b-4380-b6cc-d657ad987d74-tls-certs\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.339350 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.339319    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64ck\" (UniqueName: \"kubernetes.io/projected/00311ce8-747b-4380-b6cc-d657ad987d74-kube-api-access-h64ck\") pod \"explicit-model-0066f8-kserve-648cfc5465-jmn97\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") " pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.417558 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.417473    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:02.691870 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:02.691844    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"]
Sep 09 21:08:02.693911 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:08:02.693875    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00311ce8_747b_4380_b6cc_d657ad987d74.slice/crio-328be4780d86feefab83b0d881458d7704d8d77fc139158eef62aae9304ab9fd WatchSource:0}: Error finding container 328be4780d86feefab83b0d881458d7704d8d77fc139158eef62aae9304ab9fd: Status 404 returned error can't find the container with id 328be4780d86feefab83b0d881458d7704d8d77fc139158eef62aae9304ab9fd
Sep 09 21:08:03.126164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:03.126063    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97" event={"ID":"00311ce8-747b-4380-b6cc-d657ad987d74","Type":"ContainerStarted","Data":"2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd"}
Sep 09 21:08:03.126164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:03.126111    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97" event={"ID":"00311ce8-747b-4380-b6cc-d657ad987d74","Type":"ContainerStarted","Data":"328be4780d86feefab83b0d881458d7704d8d77fc139158eef62aae9304ab9fd"}
Sep 09 21:08:03.143770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:03.143702    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97" podStartSLOduration=1.143686474 podStartE2EDuration="1.143686474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:03.143108804 +0000 UTC m=+1171.645938804" watchObservedRunningTime="2026-09-09 21:08:03.143686474 +0000 UTC m=+1171.646516475"
Sep 09 21:08:10.893953 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:10.893912    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"]
Sep 09 21:08:12.418447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:12.418417    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:25.942291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:25.942248    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97" podUID="00311ce8-747b-4380-b6cc-d657ad987d74" containerName="main" containerID="cri-o://2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd" gracePeriod=45
Sep 09 21:08:26.197235 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.197177    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:26.247876 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.247840    2548 generic.go:356] "Generic (PLEG): container finished" podID="00311ce8-747b-4380-b6cc-d657ad987d74" containerID="2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd" exitCode=0
Sep 09 21:08:26.248007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.247891    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"
Sep 09 21:08:26.248007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.247913    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97" event={"ID":"00311ce8-747b-4380-b6cc-d657ad987d74","Type":"ContainerDied","Data":"2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd"}
Sep 09 21:08:26.248007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.247961    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97" event={"ID":"00311ce8-747b-4380-b6cc-d657ad987d74","Type":"ContainerDied","Data":"328be4780d86feefab83b0d881458d7704d8d77fc139158eef62aae9304ab9fd"}
Sep 09 21:08:26.248007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.247985    2548 scope.go:117] "RemoveContainer" containerID="2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd"
Sep 09 21:08:26.256949 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.256931    2548 scope.go:117] "RemoveContainer" containerID="2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd"
Sep 09 21:08:26.257202 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:08:26.257182    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd\": container with ID starting with 2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd not found: ID does not exist" containerID="2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd"
Sep 09 21:08:26.257249 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.257213    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd"} err="failed to get container status \"2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd\": rpc error: code = NotFound desc = could not find container \"2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd\": container with ID starting with 2f337f905576d5b02179f8719339d7da8fd3f96e493792f8de3679bf800b38bd not found: ID does not exist"
Sep 09 21:08:26.276281 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276260    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00311ce8-747b-4380-b6cc-d657ad987d74-tls-certs\") pod \"00311ce8-747b-4380-b6cc-d657ad987d74\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") "
Sep 09 21:08:26.276378 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276329    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-dshm\") pod \"00311ce8-747b-4380-b6cc-d657ad987d74\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") "
Sep 09 21:08:26.276378 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276346    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-home\") pod \"00311ce8-747b-4380-b6cc-d657ad987d74\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") "
Sep 09 21:08:26.276378 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276367    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-model-cache\") pod \"00311ce8-747b-4380-b6cc-d657ad987d74\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") "
Sep 09 21:08:26.276523 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276412    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h64ck\" (UniqueName: \"kubernetes.io/projected/00311ce8-747b-4380-b6cc-d657ad987d74-kube-api-access-h64ck\") pod \"00311ce8-747b-4380-b6cc-d657ad987d74\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") "
Sep 09 21:08:26.276523 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276470    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-tmp-dir\") pod \"00311ce8-747b-4380-b6cc-d657ad987d74\" (UID: \"00311ce8-747b-4380-b6cc-d657ad987d74\") "
Sep 09 21:08:26.276664 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276644    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-home" (OuterVolumeSpecName: "home") pod "00311ce8-747b-4380-b6cc-d657ad987d74" (UID: "00311ce8-747b-4380-b6cc-d657ad987d74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:26.276728 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276660    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-model-cache" (OuterVolumeSpecName: "model-cache") pod "00311ce8-747b-4380-b6cc-d657ad987d74" (UID: "00311ce8-747b-4380-b6cc-d657ad987d74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:26.276847 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276822    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-home\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:08:26.276847 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276847    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-model-cache\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:08:26.277008 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.276847    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00311ce8-747b-4380-b6cc-d657ad987d74" (UID: "00311ce8-747b-4380-b6cc-d657ad987d74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:26.278442 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.278418    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00311ce8-747b-4380-b6cc-d657ad987d74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00311ce8-747b-4380-b6cc-d657ad987d74" (UID: "00311ce8-747b-4380-b6cc-d657ad987d74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:26.278536 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.278475    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-dshm" (OuterVolumeSpecName: "dshm") pod "00311ce8-747b-4380-b6cc-d657ad987d74" (UID: "00311ce8-747b-4380-b6cc-d657ad987d74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:26.278536 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.278500    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00311ce8-747b-4380-b6cc-d657ad987d74-kube-api-access-h64ck" (OuterVolumeSpecName: "kube-api-access-h64ck") pod "00311ce8-747b-4380-b6cc-d657ad987d74" (UID: "00311ce8-747b-4380-b6cc-d657ad987d74"). InnerVolumeSpecName "kube-api-access-h64ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:26.377658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.377632    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64ck\" (UniqueName: \"kubernetes.io/projected/00311ce8-747b-4380-b6cc-d657ad987d74-kube-api-access-h64ck\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:08:26.377658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.377658    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-tmp-dir\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:08:26.377827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.377669    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00311ce8-747b-4380-b6cc-d657ad987d74-tls-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:08:26.377827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.377678    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00311ce8-747b-4380-b6cc-d657ad987d74-dshm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:08:26.572235 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.572199    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"]
Sep 09 21:08:26.575702 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:26.575670    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0066f8/explicit-model-0066f8-kserve-648cfc5465-jmn97"]
Sep 09 21:08:27.968597 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:27.968562    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00311ce8-747b-4380-b6cc-d657ad987d74" path="/var/lib/kubelet/pods/00311ce8-747b-4380-b6cc-d657ad987d74/volumes"
Sep 09 21:08:32.007927 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:32.007893    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:08:32.012452 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:32.012429    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:08:52.007315 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.007267    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"]
Sep 09 21:08:52.008015 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.007995    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00311ce8-747b-4380-b6cc-d657ad987d74" containerName="main"
Sep 09 21:08:52.008073 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.008017    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="00311ce8-747b-4380-b6cc-d657ad987d74" containerName="main"
Sep 09 21:08:52.008111 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.008092    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="00311ce8-747b-4380-b6cc-d657ad987d74" containerName="main"
Sep 09 21:08:52.008111 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.008100    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98bc48c-b65f-4c4d-890b-dc68b843ad15" containerName="istio-proxy"
Sep 09 21:08:52.012397 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.012379    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.015288 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.015255    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-46ba3a\"/\"kube-root-ca.crt\""
Sep 09 21:08:52.015415 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.015302    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-46ba3a\"/\"openshift-service-ca.crt\""
Sep 09 21:08:52.015415 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.015257    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-46ba3a\"/\"default-dockercfg-tz8w4\""
Sep 09 21:08:52.015415 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.015257    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-46ba3a\"/\"noten-model-46ba3a-kserve-self-signed-certs\""
Sep 09 21:08:52.020238 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.020218    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"]
Sep 09 21:08:52.136666 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.136628    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d445a4-a142-488d-ba8b-e7b3d145857b-tls-certs\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.136845 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.136681    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-dshm\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.136845 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.136728    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-home\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.136845 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.136803    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-model-cache\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.137009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.136857    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-tmp-dir\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.137009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.136875    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbt4v\" (UniqueName: \"kubernetes.io/projected/31d445a4-a142-488d-ba8b-e7b3d145857b-kube-api-access-fbt4v\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237388    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-home\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237452    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-model-cache\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237493    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-tmp-dir\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237510    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt4v\" (UniqueName: \"kubernetes.io/projected/31d445a4-a142-488d-ba8b-e7b3d145857b-kube-api-access-fbt4v\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237627 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237589    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d445a4-a142-488d-ba8b-e7b3d145857b-tls-certs\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237891 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237632    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-dshm\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237891 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237846    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-home\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.237891 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237870    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-model-cache\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.238010 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.237944    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-tmp-dir\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.240110 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.240084    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-dshm\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.240406 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.240386    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d445a4-a142-488d-ba8b-e7b3d145857b-tls-certs\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.246366 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.246337    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt4v\" (UniqueName: \"kubernetes.io/projected/31d445a4-a142-488d-ba8b-e7b3d145857b-kube-api-access-fbt4v\") pod \"noten-model-46ba3a-kserve-6d8d4848cf-qp969\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") " pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.322822 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.322744    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:08:52.593597 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:52.593573    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"]
Sep 09 21:08:52.595818 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:08:52.595788    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31d445a4_a142_488d_ba8b_e7b3d145857b.slice/crio-9a75b54a8f239ea74b7f32efbde878b77b14515c5c046ed4e7e25b0d051ed752 WatchSource:0}: Error finding container 9a75b54a8f239ea74b7f32efbde878b77b14515c5c046ed4e7e25b0d051ed752: Status 404 returned error can't find the container with id 9a75b54a8f239ea74b7f32efbde878b77b14515c5c046ed4e7e25b0d051ed752
Sep 09 21:08:53.400086 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:53.400048    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969" event={"ID":"31d445a4-a142-488d-ba8b-e7b3d145857b","Type":"ContainerStarted","Data":"305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07"}
Sep 09 21:08:53.400476 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:53.400095    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969" event={"ID":"31d445a4-a142-488d-ba8b-e7b3d145857b","Type":"ContainerStarted","Data":"9a75b54a8f239ea74b7f32efbde878b77b14515c5c046ed4e7e25b0d051ed752"}
Sep 09 21:08:53.417430 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:53.417374    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969" podStartSLOduration=2.417354543 podStartE2EDuration="2.417354543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:53.415881344 +0000 UTC m=+1221.918711345" watchObservedRunningTime="2026-09-09 21:08:53.417354543 +0000 UTC m=+1221.920184544"
Sep 09 21:08:55.219272 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:08:55.219238    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"]
Sep 09 21:09:02.323097 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:02.323055    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:09:07.105855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.105817    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"]
Sep 09 21:09:07.106283 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.106063    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44" podUID="4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" containerName="payload-processing" containerID="cri-o://9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6" gracePeriod=30
Sep 09 21:09:07.116531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.116013    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"]
Sep 09 21:09:07.116905 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.116874    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb" podUID="858be8e1-49e0-41a4-8ad1-0023d993bdfb" containerName="payload-pre-processing" containerID="cri-o://0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d" gracePeriod=30
Sep 09 21:09:07.404367 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.404194    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:09:07.425428 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.425397    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:09:07.478053 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.478021    2548 generic.go:356] "Generic (PLEG): container finished" podID="4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" containerID="9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6" exitCode=0
Sep 09 21:09:07.478208 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.478075    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"
Sep 09 21:09:07.478208 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.478110    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44" event={"ID":"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815","Type":"ContainerDied","Data":"9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6"}
Sep 09 21:09:07.478208 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.478175    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44" event={"ID":"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815","Type":"ContainerDied","Data":"20b6617cc16328bc153c0b63649567443526c80c41b2ebe73cf45944321a8514"}
Sep 09 21:09:07.478368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.478209    2548 scope.go:117] "RemoveContainer" containerID="9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6"
Sep 09 21:09:07.479749 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.479711    2548 generic.go:356] "Generic (PLEG): container finished" podID="858be8e1-49e0-41a4-8ad1-0023d993bdfb" containerID="0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d" exitCode=0
Sep 09 21:09:07.479846 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.479775    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb" event={"ID":"858be8e1-49e0-41a4-8ad1-0023d993bdfb","Type":"ContainerDied","Data":"0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d"}
Sep 09 21:09:07.479893 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.479846    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb" event={"ID":"858be8e1-49e0-41a4-8ad1-0023d993bdfb","Type":"ContainerDied","Data":"604015a9281ff008c054ee11950ada2f9d10ce79a5cb32dc5d57bc7456245713"}
Sep 09 21:09:07.479893 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.479780    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"
Sep 09 21:09:07.485002 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.484980    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9jbh\" (UniqueName: \"kubernetes.io/projected/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-kube-api-access-g9jbh\") pod \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") "
Sep 09 21:09:07.485120 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.485010    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858be8e1-49e0-41a4-8ad1-0023d993bdfb-plugins-config-volume\") pod \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") "
Sep 09 21:09:07.485120 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.485066    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzwp6\" (UniqueName: \"kubernetes.io/projected/858be8e1-49e0-41a4-8ad1-0023d993bdfb-kube-api-access-vzwp6\") pod \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\" (UID: \"858be8e1-49e0-41a4-8ad1-0023d993bdfb\") "
Sep 09 21:09:07.485120 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.485103    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-plugins-config-volume\") pod \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\" (UID: \"4aa59509-85d3-4fe1-9c2c-1ccf27dd8815\") "
Sep 09 21:09:07.485560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.485529    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858be8e1-49e0-41a4-8ad1-0023d993bdfb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "858be8e1-49e0-41a4-8ad1-0023d993bdfb" (UID: "858be8e1-49e0-41a4-8ad1-0023d993bdfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:07.485657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.485588    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" (UID: "4aa59509-85d3-4fe1-9c2c-1ccf27dd8815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:07.487162 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.487137    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-kube-api-access-g9jbh" (OuterVolumeSpecName: "kube-api-access-g9jbh") pod "4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" (UID: "4aa59509-85d3-4fe1-9c2c-1ccf27dd8815"). InnerVolumeSpecName "kube-api-access-g9jbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:07.487264 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.487247    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858be8e1-49e0-41a4-8ad1-0023d993bdfb-kube-api-access-vzwp6" (OuterVolumeSpecName: "kube-api-access-vzwp6") pod "858be8e1-49e0-41a4-8ad1-0023d993bdfb" (UID: "858be8e1-49e0-41a4-8ad1-0023d993bdfb"). InnerVolumeSpecName "kube-api-access-vzwp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:07.488487 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.488473    2548 scope.go:117] "RemoveContainer" containerID="9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6"
Sep 09 21:09:07.488785 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:09:07.488762    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6\": container with ID starting with 9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6 not found: ID does not exist" containerID="9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6"
Sep 09 21:09:07.488850 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.488792    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6"} err="failed to get container status \"9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6\": rpc error: code = NotFound desc = could not find container \"9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6\": container with ID starting with 9248b423fe976951c8c6a82d7da264c590ca2e9fde1d8a57d82f1dae477831f6 not found: ID does not exist"
Sep 09 21:09:07.488850 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.488808    2548 scope.go:117] "RemoveContainer" containerID="0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d"
Sep 09 21:09:07.496221 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.496203    2548 scope.go:117] "RemoveContainer" containerID="0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d"
Sep 09 21:09:07.496475 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:09:07.496457    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d\": container with ID starting with 0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d not found: ID does not exist" containerID="0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d"
Sep 09 21:09:07.496554 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.496479    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d"} err="failed to get container status \"0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d\": rpc error: code = NotFound desc = could not find container \"0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d\": container with ID starting with 0c8087a922c700f99d981eead915fff01174f3e54024dd92e88636391aba155d not found: ID does not exist"
Sep 09 21:09:07.586655 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.586623    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwp6\" (UniqueName: \"kubernetes.io/projected/858be8e1-49e0-41a4-8ad1-0023d993bdfb-kube-api-access-vzwp6\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:07.586655 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.586650    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:07.586890 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.586665    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jbh\" (UniqueName: \"kubernetes.io/projected/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815-kube-api-access-g9jbh\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:07.586890 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.586680    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/858be8e1-49e0-41a4-8ad1-0023d993bdfb-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:07.804499 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.804444    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"]
Sep 09 21:09:07.806982 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.806957    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9a54ea-688b57cb47-qmq44"]
Sep 09 21:09:07.820952 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.820925    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"]
Sep 09 21:09:07.824097 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.824073    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9a54ea-949df64d6-46bkb"]
Sep 09 21:09:07.968835 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.968802    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" path="/var/lib/kubelet/pods/4aa59509-85d3-4fe1-9c2c-1ccf27dd8815/volumes"
Sep 09 21:09:07.969217 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:07.969203    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858be8e1-49e0-41a4-8ad1-0023d993bdfb" path="/var/lib/kubelet/pods/858be8e1-49e0-41a4-8ad1-0023d993bdfb/volumes"
Sep 09 21:09:10.433405 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.433306    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969" podUID="31d445a4-a142-488d-ba8b-e7b3d145857b" containerName="main" containerID="cri-o://305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07" gracePeriod=45
Sep 09 21:09:10.672795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.672771    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:09:10.714522 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714490    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-tmp-dir\") pod \"31d445a4-a142-488d-ba8b-e7b3d145857b\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") "
Sep 09 21:09:10.714662 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714541    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-model-cache\") pod \"31d445a4-a142-488d-ba8b-e7b3d145857b\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") "
Sep 09 21:09:10.714662 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714578    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-home\") pod \"31d445a4-a142-488d-ba8b-e7b3d145857b\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") "
Sep 09 21:09:10.714662 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714608    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbt4v\" (UniqueName: \"kubernetes.io/projected/31d445a4-a142-488d-ba8b-e7b3d145857b-kube-api-access-fbt4v\") pod \"31d445a4-a142-488d-ba8b-e7b3d145857b\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") "
Sep 09 21:09:10.714856 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714696    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-dshm\") pod \"31d445a4-a142-488d-ba8b-e7b3d145857b\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") "
Sep 09 21:09:10.714856 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714774    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31d445a4-a142-488d-ba8b-e7b3d145857b" (UID: "31d445a4-a142-488d-ba8b-e7b3d145857b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:10.714856 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714843    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d445a4-a142-488d-ba8b-e7b3d145857b-tls-certs\") pod \"31d445a4-a142-488d-ba8b-e7b3d145857b\" (UID: \"31d445a4-a142-488d-ba8b-e7b3d145857b\") "
Sep 09 21:09:10.715017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714860    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-model-cache" (OuterVolumeSpecName: "model-cache") pod "31d445a4-a142-488d-ba8b-e7b3d145857b" (UID: "31d445a4-a142-488d-ba8b-e7b3d145857b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:10.715017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.714876    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-home" (OuterVolumeSpecName: "home") pod "31d445a4-a142-488d-ba8b-e7b3d145857b" (UID: "31d445a4-a142-488d-ba8b-e7b3d145857b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:10.715166 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.715148    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-home\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:10.715223 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.715171    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-tmp-dir\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:10.715223 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.715185    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-model-cache\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:10.717165 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.717129    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d445a4-a142-488d-ba8b-e7b3d145857b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31d445a4-a142-488d-ba8b-e7b3d145857b" (UID: "31d445a4-a142-488d-ba8b-e7b3d145857b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:10.717286 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.717242    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-dshm" (OuterVolumeSpecName: "dshm") pod "31d445a4-a142-488d-ba8b-e7b3d145857b" (UID: "31d445a4-a142-488d-ba8b-e7b3d145857b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:10.717286 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.717265    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d445a4-a142-488d-ba8b-e7b3d145857b-kube-api-access-fbt4v" (OuterVolumeSpecName: "kube-api-access-fbt4v") pod "31d445a4-a142-488d-ba8b-e7b3d145857b" (UID: "31d445a4-a142-488d-ba8b-e7b3d145857b"). InnerVolumeSpecName "kube-api-access-fbt4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:10.816144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.816112    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d445a4-a142-488d-ba8b-e7b3d145857b-tls-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:10.816144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.816139    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbt4v\" (UniqueName: \"kubernetes.io/projected/31d445a4-a142-488d-ba8b-e7b3d145857b-kube-api-access-fbt4v\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:10.816144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:10.816148    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d445a4-a142-488d-ba8b-e7b3d145857b-dshm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:11.504997 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.504964    2548 generic.go:356] "Generic (PLEG): container finished" podID="31d445a4-a142-488d-ba8b-e7b3d145857b" containerID="305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07" exitCode=0
Sep 09 21:09:11.505468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.505009    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"
Sep 09 21:09:11.505468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.505021    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969" event={"ID":"31d445a4-a142-488d-ba8b-e7b3d145857b","Type":"ContainerDied","Data":"305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07"}
Sep 09 21:09:11.505468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.505059    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969" event={"ID":"31d445a4-a142-488d-ba8b-e7b3d145857b","Type":"ContainerDied","Data":"9a75b54a8f239ea74b7f32efbde878b77b14515c5c046ed4e7e25b0d051ed752"}
Sep 09 21:09:11.505468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.505082    2548 scope.go:117] "RemoveContainer" containerID="305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07"
Sep 09 21:09:11.514225 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.514210    2548 scope.go:117] "RemoveContainer" containerID="305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07"
Sep 09 21:09:11.514471 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:09:11.514454    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07\": container with ID starting with 305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07 not found: ID does not exist" containerID="305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07"
Sep 09 21:09:11.514511 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.514480    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07"} err="failed to get container status \"305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07\": rpc error: code = NotFound desc = could not find container \"305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07\": container with ID starting with 305cf78a2c72cc68e9ec7c03e0e7a7b7619e31aba41d54306345f815d0405f07 not found: ID does not exist"
Sep 09 21:09:11.525918 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.525892    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"]
Sep 09 21:09:11.530084 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.530061    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-46ba3a/noten-model-46ba3a-kserve-6d8d4848cf-qp969"]
Sep 09 21:09:11.968630 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:11.968597    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d445a4-a142-488d-ba8b-e7b3d145857b" path="/var/lib/kubelet/pods/31d445a4-a142-488d-ba8b-e7b3d145857b/volumes"
Sep 09 21:09:14.019973 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:14.019932    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"]
Sep 09 21:09:14.020424 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:14.020195    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy" containerID="cri-o://b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a" gracePeriod=30
Sep 09 21:09:19.282462 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.282408    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:09:19.391607 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391578    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-data\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391788 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391629    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg6kd\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-kube-api-access-fg6kd\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391788 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391684    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310a117b-3658-4f4a-82e2-539629e0cfad-istiod-ca-cert\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391788 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391768    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-socket\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391822    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-credential-socket\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391845    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-data" (OuterVolumeSpecName: "istio-data") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:19.391972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391854    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310a117b-3658-4f4a-82e2-539629e0cfad-istio-podinfo\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391917    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-certs\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.391972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.391965    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-envoy\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.392226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392040    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-istio-token\") pod \"310a117b-3658-4f4a-82e2-539629e0cfad\" (UID: \"310a117b-3658-4f4a-82e2-539629e0cfad\") "
Sep 09 21:09:19.392226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392066    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:19.392226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392116    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:19.392226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392192    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310a117b-3658-4f4a-82e2-539629e0cfad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:19.392226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392205    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:19.392451 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392428    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310a117b-3658-4f4a-82e2-539629e0cfad-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.392531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392454    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.392531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392468    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.392531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392483    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.392531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.392496    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.393976 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.393944    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-kube-api-access-fg6kd" (OuterVolumeSpecName: "kube-api-access-fg6kd") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "kube-api-access-fg6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:19.394074 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.394045    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/310a117b-3658-4f4a-82e2-539629e0cfad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:09:19.394180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.394164    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-istio-token" (OuterVolumeSpecName: "istio-token") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:19.394302 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.394288    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "310a117b-3658-4f4a-82e2-539629e0cfad" (UID: "310a117b-3658-4f4a-82e2-539629e0cfad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:19.493033 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.493008    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310a117b-3658-4f4a-82e2-539629e0cfad-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.493033 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.493030    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310a117b-3658-4f4a-82e2-539629e0cfad-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.493156 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.493039    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.493156 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.493047    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6kd\" (UniqueName: \"kubernetes.io/projected/310a117b-3658-4f4a-82e2-539629e0cfad-kube-api-access-fg6kd\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:19.550080 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.550021    2548 generic.go:356] "Generic (PLEG): container finished" podID="310a117b-3658-4f4a-82e2-539629e0cfad" containerID="b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a" exitCode=0
Sep 09 21:09:19.550166 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.550080    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" event={"ID":"310a117b-3658-4f4a-82e2-539629e0cfad","Type":"ContainerDied","Data":"b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a"}
Sep 09 21:09:19.550166 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.550087    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"
Sep 09 21:09:19.550166 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.550122    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d" event={"ID":"310a117b-3658-4f4a-82e2-539629e0cfad","Type":"ContainerDied","Data":"0ad0f3585eb1a6a0c3da3c82e814f424ad32d52139e94d2b9e6bba978068e8f0"}
Sep 09 21:09:19.550166 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.550144    2548 scope.go:117] "RemoveContainer" containerID="b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a"
Sep 09 21:09:19.559445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.559428    2548 scope.go:117] "RemoveContainer" containerID="d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"
Sep 09 21:09:19.567366 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.567345    2548 scope.go:117] "RemoveContainer" containerID="b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a"
Sep 09 21:09:19.567622 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:09:19.567601    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a\": container with ID starting with b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a not found: ID does not exist" containerID="b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a"
Sep 09 21:09:19.567672 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.567630    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a"} err="failed to get container status \"b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a\": rpc error: code = NotFound desc = could not find container \"b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a\": container with ID starting with b9ea7139e395b054b6309e46e27ec5186f8935e02b19c0422a8a3aba2680a21a not found: ID does not exist"
Sep 09 21:09:19.567672 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.567646    2548 scope.go:117] "RemoveContainer" containerID="d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"
Sep 09 21:09:19.567934 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:09:19.567914    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0\": container with ID starting with d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0 not found: ID does not exist" containerID="d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"
Sep 09 21:09:19.568003 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.567945    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0"} err="failed to get container status \"d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0\": rpc error: code = NotFound desc = could not find container \"d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0\": container with ID starting with d821c03530c9f41fac2c55152774619a34eb7161246d86ffc77758ff6fe651f0 not found: ID does not exist"
Sep 09 21:09:19.572179 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.572156    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"]
Sep 09 21:09:19.575582 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.575561    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9a54ea-openshift-default-c774f7d5b-vnz9d"]
Sep 09 21:09:19.968221 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:19.968193    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" path="/var/lib/kubelet/pods/310a117b-3658-4f4a-82e2-539629e0cfad/volumes"
Sep 09 21:09:24.992264 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:24.992191    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"]
Sep 09 21:09:24.992651 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:24.992417    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" podUID="543dd57a-99f1-4090-9247-69420faa2b95" containerName="maas-api" containerID="cri-o://079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8" gracePeriod=30
Sep 09 21:09:25.251355 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.251285    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:09:25.343962 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.343936    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-metrics-certs\") pod \"543dd57a-99f1-4090-9247-69420faa2b95\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") "
Sep 09 21:09:25.344118 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.343991    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls\") pod \"543dd57a-99f1-4090-9247-69420faa2b95\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") "
Sep 09 21:09:25.344118 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.344017    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543dd57a-99f1-4090-9247-69420faa2b95-openshift-service-ca\") pod \"543dd57a-99f1-4090-9247-69420faa2b95\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") "
Sep 09 21:09:25.344118 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.344057    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb256\" (UniqueName: \"kubernetes.io/projected/543dd57a-99f1-4090-9247-69420faa2b95-kube-api-access-tb256\") pod \"543dd57a-99f1-4090-9247-69420faa2b95\" (UID: \"543dd57a-99f1-4090-9247-69420faa2b95\") "
Sep 09 21:09:25.344418 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.344389    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543dd57a-99f1-4090-9247-69420faa2b95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "543dd57a-99f1-4090-9247-69420faa2b95" (UID: "543dd57a-99f1-4090-9247-69420faa2b95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:25.346279 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.346242    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543dd57a-99f1-4090-9247-69420faa2b95-kube-api-access-tb256" (OuterVolumeSpecName: "kube-api-access-tb256") pod "543dd57a-99f1-4090-9247-69420faa2b95" (UID: "543dd57a-99f1-4090-9247-69420faa2b95"). InnerVolumeSpecName "kube-api-access-tb256". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:25.346435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.346292    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "543dd57a-99f1-4090-9247-69420faa2b95" (UID: "543dd57a-99f1-4090-9247-69420faa2b95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:25.346435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.346343    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "543dd57a-99f1-4090-9247-69420faa2b95" (UID: "543dd57a-99f1-4090-9247-69420faa2b95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:25.444993 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.444962    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:25.444993 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.444986    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/543dd57a-99f1-4090-9247-69420faa2b95-maas-api-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:25.444993 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.444996    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/543dd57a-99f1-4090-9247-69420faa2b95-openshift-service-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:25.445266 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.445005    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb256\" (UniqueName: \"kubernetes.io/projected/543dd57a-99f1-4090-9247-69420faa2b95-kube-api-access-tb256\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:09:25.582707 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.582624    2548 generic.go:356] "Generic (PLEG): container finished" podID="543dd57a-99f1-4090-9247-69420faa2b95" containerID="079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8" exitCode=0
Sep 09 21:09:25.582707 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.582663    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"
Sep 09 21:09:25.582707 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.582692    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" event={"ID":"543dd57a-99f1-4090-9247-69420faa2b95","Type":"ContainerDied","Data":"079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8"}
Sep 09 21:09:25.582951 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.582746    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw" event={"ID":"543dd57a-99f1-4090-9247-69420faa2b95","Type":"ContainerDied","Data":"8de7fdac9250c166a344401b18f1209ef79f4c6e60eba84f40d117c65ba6f53c"}
Sep 09 21:09:25.582951 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.582768    2548 scope.go:117] "RemoveContainer" containerID="079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8"
Sep 09 21:09:25.592461 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.592443    2548 scope.go:117] "RemoveContainer" containerID="079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8"
Sep 09 21:09:25.592754 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:09:25.592716    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8\": container with ID starting with 079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8 not found: ID does not exist" containerID="079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8"
Sep 09 21:09:25.592852 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.592761    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8"} err="failed to get container status \"079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8\": rpc error: code = NotFound desc = could not find container \"079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8\": container with ID starting with 079b60307861ef729271a93e09709508042ba1ae10847dd2399dc4f46ec509f8 not found: ID does not exist"
Sep 09 21:09:25.616139 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.616115    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"]
Sep 09 21:09:25.622855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.622833    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-98dad6-5878b568c-6kgtw"]
Sep 09 21:09:25.968338 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:25.968311    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543dd57a-99f1-4090-9247-69420faa2b95" path="/var/lib/kubelet/pods/543dd57a-99f1-4090-9247-69420faa2b95/volumes"
Sep 09 21:09:33.535502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.535460    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"]
Sep 09 21:09:33.536052 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536031    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543dd57a-99f1-4090-9247-69420faa2b95" containerName="maas-api"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536053    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="543dd57a-99f1-4090-9247-69420faa2b95" containerName="maas-api"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536090    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536099    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536115    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" containerName="payload-processing"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536125    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" containerName="payload-processing"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536142    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858be8e1-49e0-41a4-8ad1-0023d993bdfb" containerName="payload-pre-processing"
Sep 09 21:09:33.536152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536151    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="858be8e1-49e0-41a4-8ad1-0023d993bdfb" containerName="payload-pre-processing"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536163    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d445a4-a142-488d-ba8b-e7b3d145857b" containerName="main"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536172    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d445a4-a142-488d-ba8b-e7b3d145857b" containerName="main"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536192    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536200    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536302    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536315    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa59509-85d3-4fe1-9c2c-1ccf27dd8815" containerName="payload-processing"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536325    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d445a4-a142-488d-ba8b-e7b3d145857b" containerName="main"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536335    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="543dd57a-99f1-4090-9247-69420faa2b95" containerName="maas-api"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536345    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="310a117b-3658-4f4a-82e2-539629e0cfad" containerName="istio-proxy"
Sep 09 21:09:33.536483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.536355    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="858be8e1-49e0-41a4-8ad1-0023d993bdfb" containerName="payload-pre-processing"
Sep 09 21:09:33.541258 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.541235    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.545759 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.545511    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3d3cac-openshift-default-dockercfg-lmshs\""
Sep 09 21:09:33.560045 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.560023    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"]
Sep 09 21:09:33.718849 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.718813    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b297dfb-20a2-4263-a857-52e889c86c69-istiod-ca-cert\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.718849 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.718850    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b297dfb-20a2-4263-a857-52e889c86c69-istio-podinfo\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719062 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.718889    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-credential-socket\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719062 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.718909    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-socket\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719062 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.718929    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-data\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.719063    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkh2\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-kube-api-access-8gkh2\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.719106    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-istio-token\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.719135    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-certs\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.719180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.719156    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-envoy\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.819955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.819872    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b297dfb-20a2-4263-a857-52e889c86c69-istiod-ca-cert\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.819955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.819912    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b297dfb-20a2-4263-a857-52e889c86c69-istio-podinfo\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820134 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820036    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-credential-socket\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820134 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820074    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-socket\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820134 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820104    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-data\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820230    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkh2\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-kube-api-access-8gkh2\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820427 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820404    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-credential-socket\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820505 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820456    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-socket\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820505 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820484    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-istio-token\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820505 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820495    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-data\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820538    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-certs\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820572    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-envoy\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820608    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b297dfb-20a2-4263-a857-52e889c86c69-istiod-ca-cert\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.820864 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.820846    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-certs\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.822460 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.822438    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b297dfb-20a2-4263-a857-52e889c86c69-istio-podinfo\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.822829 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.822806    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-envoy\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.832213 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.832184    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-istio-token\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.832351 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.832297    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkh2\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-kube-api-access-8gkh2\") pod \"e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") " pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:33.851814 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:33.851790    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:34.130352 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.130325    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"]
Sep 09 21:09:34.131285 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:09:34.131255    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b297dfb_20a2_4263_a857_52e889c86c69.slice/crio-c31cee9586df073edd5de02cba93dd7f85aaa0027833683c87795f2a7b8def9c WatchSource:0}: Error finding container c31cee9586df073edd5de02cba93dd7f85aaa0027833683c87795f2a7b8def9c: Status 404 returned error can't find the container with id c31cee9586df073edd5de02cba93dd7f85aaa0027833683c87795f2a7b8def9c
Sep 09 21:09:34.133408 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.133380    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:09:34.133469 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.133443    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:09:34.133507 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.133473    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:09:34.634220 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.634185    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" event={"ID":"9b297dfb-20a2-4263-a857-52e889c86c69","Type":"ContainerStarted","Data":"fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"}
Sep 09 21:09:34.634641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.634244    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" event={"ID":"9b297dfb-20a2-4263-a857-52e889c86c69","Type":"ContainerStarted","Data":"c31cee9586df073edd5de02cba93dd7f85aaa0027833683c87795f2a7b8def9c"}
Sep 09 21:09:34.656584 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.656522    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podStartSLOduration=1.656507581 podStartE2EDuration="1.656507581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:34.654246041 +0000 UTC m=+1263.157076036" watchObservedRunningTime="2026-09-09 21:09:34.656507581 +0000 UTC m=+1263.159337581"
Sep 09 21:09:34.852576 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:34.852543    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:35.858239 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:35.858202    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:36.643943 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:36.643897    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:36.644847 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:36.644824    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:37.822368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.822335    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"]
Sep 09 21:09:37.826807 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.826787    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:37.828975 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.828950    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3d3cac-serving-cert\""
Sep 09 21:09:37.839449 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.839430    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"]
Sep 09 21:09:37.957968 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.957932    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:37.958135 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.957977    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:37.958135 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.958035    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:37.958135 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:37.958094    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qrk\" (UniqueName: \"kubernetes.io/projected/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-kube-api-access-d6qrk\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.059412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.059371    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qrk\" (UniqueName: \"kubernetes.io/projected/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-kube-api-access-d6qrk\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.059602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.059482    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.059602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.059511    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.059602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.059545    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.060223 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.060193    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.061891 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.061870    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.061997 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.061940    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.068527 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.068507    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qrk\" (UniqueName: \"kubernetes.io/projected/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-kube-api-access-d6qrk\") pod \"maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.139028 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.138970    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:38.421231 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.421203    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"]
Sep 09 21:09:38.424029 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:09:38.424000    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7020b2d_54f1_4f85_ae7c_9dbd1895971e.slice/crio-2138ba9f216c09107612c8d21f2371b2d883ffa52eb6afeb2a0192fd227dc573 WatchSource:0}: Error finding container 2138ba9f216c09107612c8d21f2371b2d883ffa52eb6afeb2a0192fd227dc573: Status 404 returned error can't find the container with id 2138ba9f216c09107612c8d21f2371b2d883ffa52eb6afeb2a0192fd227dc573
Sep 09 21:09:38.477424 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.477394    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"]
Sep 09 21:09:38.482444 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.482421    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.484615 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.484595    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3d3cac\""
Sep 09 21:09:38.493758 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.493705    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"]
Sep 09 21:09:38.655878 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.655833    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt" event={"ID":"c7020b2d-54f1-4f85-ae7c-9dbd1895971e","Type":"ContainerStarted","Data":"2138ba9f216c09107612c8d21f2371b2d883ffa52eb6afeb2a0192fd227dc573"}
Sep 09 21:09:38.664526 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.664494    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll969\" (UniqueName: \"kubernetes.io/projected/4f875000-bff9-4b09-ace1-1f9a54f357da-kube-api-access-ll969\") pod \"payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.664667 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.664573    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f875000-bff9-4b09-ace1-1f9a54f357da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.765169 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.765138    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll969\" (UniqueName: \"kubernetes.io/projected/4f875000-bff9-4b09-ace1-1f9a54f357da-kube-api-access-ll969\") pod \"payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.765301 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.765276    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f875000-bff9-4b09-ace1-1f9a54f357da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.766051 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.766028    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f875000-bff9-4b09-ace1-1f9a54f357da-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.777149 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.777129    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll969\" (UniqueName: \"kubernetes.io/projected/4f875000-bff9-4b09-ace1-1f9a54f357da-kube-api-access-ll969\") pod \"payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:38.793704 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:38.793683    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:39.083442 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.083417    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"]
Sep 09 21:09:39.085012 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:09:39.084973    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f875000_bff9_4b09_ace1_1f9a54f357da.slice/crio-b6066ef07da5028953fe03c8a1ac65a8a261697d9246e048549962399102c9d8 WatchSource:0}: Error finding container b6066ef07da5028953fe03c8a1ac65a8a261697d9246e048549962399102c9d8: Status 404 returned error can't find the container with id b6066ef07da5028953fe03c8a1ac65a8a261697d9246e048549962399102c9d8
Sep 09 21:09:39.670670 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.670630    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt" event={"ID":"c7020b2d-54f1-4f85-ae7c-9dbd1895971e","Type":"ContainerStarted","Data":"cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368"}
Sep 09 21:09:39.670898 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.670747    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:39.672211 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.672185    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg" event={"ID":"4f875000-bff9-4b09-ace1-1f9a54f357da","Type":"ContainerStarted","Data":"c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1"}
Sep 09 21:09:39.672326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.672224    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg" event={"ID":"4f875000-bff9-4b09-ace1-1f9a54f357da","Type":"ContainerStarted","Data":"b6066ef07da5028953fe03c8a1ac65a8a261697d9246e048549962399102c9d8"}
Sep 09 21:09:39.672326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.672312    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:39.692445 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.692387    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt" podStartSLOduration=2.380989454 podStartE2EDuration="2.692368968s" totalImagesPullingTime="311.379514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:39.689992766 +0000 UTC m=+1268.192822778" watchObservedRunningTime="2026-09-09 21:09:39.692368968 +0000 UTC m=+1268.195198968"
Sep 09 21:09:39.708099 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:39.708049    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg" podStartSLOduration=1.70803393 podStartE2EDuration="1.70803393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:39.707497252 +0000 UTC m=+1268.210327252" watchObservedRunningTime="2026-09-09 21:09:39.70803393 +0000 UTC m=+1268.210863930"
Sep 09 21:09:45.684576 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:45.684543    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:09:50.681160 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:50.681119    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:09:51.646040 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.645990    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:51.646267 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.646079    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:51.739502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.739419    2548 generic.go:356] "Generic (PLEG): container finished" podID="9b297dfb-20a2-4263-a857-52e889c86c69" containerID="fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255" exitCode=0
Sep 09 21:09:51.739881 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.739494    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" event={"ID":"9b297dfb-20a2-4263-a857-52e889c86c69","Type":"ContainerDied","Data":"fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"}
Sep 09 21:09:51.739967 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.739953    2548 scope.go:117] "RemoveContainer" containerID="fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"
Sep 09 21:09:51.740990 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.740963    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:09:51.741077 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.741036    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:09:51.741121 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:51.741072    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:09:52.748300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:52.748252    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" event={"ID":"9b297dfb-20a2-4263-a857-52e889c86c69","Type":"ContainerStarted","Data":"5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1"}
Sep 09 21:09:52.852599 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:52.852554    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:52.852822 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:52.852689    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:52.852822 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:52.852776    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:53.852963 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:53.852501    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:53.852963 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:53.852563    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:54.853020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:54.852562    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:54.853020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:54.852630    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:55.853961 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:55.853918    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:55.854480 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:55.853987    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:56.853653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:56.853221    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:56.853653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:56.853291    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:57.852690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:57.852645    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 09 21:09:57.853295 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:57.853258    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 09 21:09:58.856236 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:58.856202    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:58.856708 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:58.856596    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:09:58.857334 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:09:58.857315    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:10:15.978079 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:15.978042    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"]
Sep 09 21:10:15.982905 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:15.982880    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:15.988329 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:15.988279    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-629537-dockercfg-644j9\""
Sep 09 21:10:15.988798 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:15.988775    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-629537\""
Sep 09 21:10:15.995475 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:15.995446    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"]
Sep 09 21:10:16.039226 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.039189    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22p5\" (UniqueName: \"kubernetes.io/projected/5142e403-852d-4156-ac74-0ceb369e8aff-kube-api-access-m22p5\") pod \"payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.039409 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.039315    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5142e403-852d-4156-ac74-0ceb369e8aff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.140629 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.140594    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5142e403-852d-4156-ac74-0ceb369e8aff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.140810 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.140654    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22p5\" (UniqueName: \"kubernetes.io/projected/5142e403-852d-4156-ac74-0ceb369e8aff-kube-api-access-m22p5\") pod \"payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.141320 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.141298    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5142e403-852d-4156-ac74-0ceb369e8aff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.156639 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.156601    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22p5\" (UniqueName: \"kubernetes.io/projected/5142e403-852d-4156-ac74-0ceb369e8aff-kube-api-access-m22p5\") pod \"payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.295134 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.295038    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.579349 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.579248    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"]
Sep 09 21:10:16.582674 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:10:16.582641    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5142e403_852d_4156_ac74_0ceb369e8aff.slice/crio-f6120b34b346c83d82f3cf98c7007d3b48379095189334c903deb454c0f5b648 WatchSource:0}: Error finding container f6120b34b346c83d82f3cf98c7007d3b48379095189334c903deb454c0f5b648: Status 404 returned error can't find the container with id f6120b34b346c83d82f3cf98c7007d3b48379095189334c903deb454c0f5b648
Sep 09 21:10:16.897837 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.897718    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2" event={"ID":"5142e403-852d-4156-ac74-0ceb369e8aff","Type":"ContainerStarted","Data":"319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337"}
Sep 09 21:10:16.897837 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.897814    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2" event={"ID":"5142e403-852d-4156-ac74-0ceb369e8aff","Type":"ContainerStarted","Data":"f6120b34b346c83d82f3cf98c7007d3b48379095189334c903deb454c0f5b648"}
Sep 09 21:10:16.898056 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.897990    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:16.917919 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:16.917863    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2" podStartSLOduration=1.91784666 podStartE2EDuration="1.91784666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:16.916688586 +0000 UTC m=+1305.419518588" watchObservedRunningTime="2026-09-09 21:10:16.91784666 +0000 UTC m=+1305.420676658"
Sep 09 21:10:27.905395 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:27.905363    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:10:44.201518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.201474    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"]
Sep 09 21:10:44.206256 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.206226    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.208461 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.208435    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-459158-serving-cert\""
Sep 09 21:10:44.217298 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.217252    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"]
Sep 09 21:10:44.309171 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.309128    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b8c94c-3016-420a-807b-d1106f3a8504-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.309458 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.309427    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.309606 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.309473    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqsw8\" (UniqueName: \"kubernetes.io/projected/a3b8c94c-3016-420a-807b-d1106f3a8504-kube-api-access-pqsw8\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.309606 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.309526    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.410882 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.410841    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b8c94c-3016-420a-807b-d1106f3a8504-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.410882 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.410888    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.411174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.410908    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqsw8\" (UniqueName: \"kubernetes.io/projected/a3b8c94c-3016-420a-807b-d1106f3a8504-kube-api-access-pqsw8\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.411174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.410937    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.411578 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.411549    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b8c94c-3016-420a-807b-d1106f3a8504-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.413422 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.413394    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.413514 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.413463    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.425939 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.425913    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqsw8\" (UniqueName: \"kubernetes.io/projected/a3b8c94c-3016-420a-807b-d1106f3a8504-kube-api-access-pqsw8\") pod \"maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.518569 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.518480    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:44.796771 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.796711    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"]
Sep 09 21:10:44.798838 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:10:44.798810    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b8c94c_3016_420a_807b_d1106f3a8504.slice/crio-8d9f1ca44d4fa2e87c6a90a5bb805a15f0deb6bdf09b42b16f7e4501159613be WatchSource:0}: Error finding container 8d9f1ca44d4fa2e87c6a90a5bb805a15f0deb6bdf09b42b16f7e4501159613be: Status 404 returned error can't find the container with id 8d9f1ca44d4fa2e87c6a90a5bb805a15f0deb6bdf09b42b16f7e4501159613be
Sep 09 21:10:44.800364 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:44.800346    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:10:45.014048 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.014013    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"]
Sep 09 21:10:45.018540 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.018518    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.021355 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.021330    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-459158-dockercfg-n6z7h\""
Sep 09 21:10:45.021480 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.021467    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-459158\""
Sep 09 21:10:45.028920 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.028895    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"]
Sep 09 21:10:45.041013 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.040974    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz" event={"ID":"a3b8c94c-3016-420a-807b-d1106f3a8504","Type":"ContainerStarted","Data":"8d9f1ca44d4fa2e87c6a90a5bb805a15f0deb6bdf09b42b16f7e4501159613be"}
Sep 09 21:10:45.118518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.118429    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpsh\" (UniqueName: \"kubernetes.io/projected/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-kube-api-access-fqpsh\") pod \"payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.118518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.118494    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.220066 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.220029    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpsh\" (UniqueName: \"kubernetes.io/projected/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-kube-api-access-fqpsh\") pod \"payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.220468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.220119    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.220786 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.220763    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.230625 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.230605    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpsh\" (UniqueName: \"kubernetes.io/projected/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-kube-api-access-fqpsh\") pod \"payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.330861 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.330821    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:45.604266 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:45.604232    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"]
Sep 09 21:10:45.606282 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:10:45.606246    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d86c3f3_39dc_403e_bb23_c4659cbcce4c.slice/crio-3b76cd1c133cffb2586d4504c6057add9d9d81dbaaee2c742805d9e5416656c2 WatchSource:0}: Error finding container 3b76cd1c133cffb2586d4504c6057add9d9d81dbaaee2c742805d9e5416656c2: Status 404 returned error can't find the container with id 3b76cd1c133cffb2586d4504c6057add9d9d81dbaaee2c742805d9e5416656c2
Sep 09 21:10:46.050064 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.050021    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz" event={"ID":"a3b8c94c-3016-420a-807b-d1106f3a8504","Type":"ContainerStarted","Data":"87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46"}
Sep 09 21:10:46.050272 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.050238    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:46.052129 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.052096    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl" event={"ID":"4d86c3f3-39dc-403e-bb23-c4659cbcce4c","Type":"ContainerStarted","Data":"f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428"}
Sep 09 21:10:46.052258 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.052146    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl" event={"ID":"4d86c3f3-39dc-403e-bb23-c4659cbcce4c","Type":"ContainerStarted","Data":"3b76cd1c133cffb2586d4504c6057add9d9d81dbaaee2c742805d9e5416656c2"}
Sep 09 21:10:46.052258 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.052191    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:10:46.067193 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.067140    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz" podStartSLOduration=1.701638698 podStartE2EDuration="2.067122271s" totalImagesPullingTime="365.483573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:46.066802346 +0000 UTC m=+1334.569632348" watchObservedRunningTime="2026-09-09 21:10:46.067122271 +0000 UTC m=+1334.569952274"
Sep 09 21:10:46.082485 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:46.082441    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl" podStartSLOduration=2.082427238 podStartE2EDuration="2.082427238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:46.081413179 +0000 UTC m=+1334.584243181" watchObservedRunningTime="2026-09-09 21:10:46.082427238 +0000 UTC m=+1334.585257238"
Sep 09 21:10:52.064658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:52.064614    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:10:57.061317 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:10:57.061240    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:11:22.081289 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.081256    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"]
Sep 09 21:11:22.085781 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.085761    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.088508 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.088488    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8b5626b-dockercfg-nbk8t\""
Sep 09 21:11:22.088921 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.088906    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8b5626b\""
Sep 09 21:11:22.096703 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.096680    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"]
Sep 09 21:11:22.196012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.195957    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c085be-1c20-4582-b15c-20d62f3884ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.196264 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.196206    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbnb\" (UniqueName: \"kubernetes.io/projected/b4c085be-1c20-4582-b15c-20d62f3884ca-kube-api-access-kcbnb\") pod \"payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.296801 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.296763    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbnb\" (UniqueName: \"kubernetes.io/projected/b4c085be-1c20-4582-b15c-20d62f3884ca-kube-api-access-kcbnb\") pod \"payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.296978 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.296879    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c085be-1c20-4582-b15c-20d62f3884ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.297600 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.297575    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c085be-1c20-4582-b15c-20d62f3884ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.304237 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.304213    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbnb\" (UniqueName: \"kubernetes.io/projected/b4c085be-1c20-4582-b15c-20d62f3884ca-kube-api-access-kcbnb\") pod \"payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.397493 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.397415    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:22.672244 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:22.672219    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"]
Sep 09 21:11:22.675470 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:11:22.675403    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c085be_1c20_4582_b15c_20d62f3884ca.slice/crio-a7336b6ed7c65a9170ce804aa2a059e85e965499115bc7eb1e1b634c71720727 WatchSource:0}: Error finding container a7336b6ed7c65a9170ce804aa2a059e85e965499115bc7eb1e1b634c71720727: Status 404 returned error can't find the container with id a7336b6ed7c65a9170ce804aa2a059e85e965499115bc7eb1e1b634c71720727
Sep 09 21:11:23.254293 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.254248    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h" event={"ID":"b4c085be-1c20-4582-b15c-20d62f3884ca","Type":"ContainerStarted","Data":"7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3"}
Sep 09 21:11:23.254781 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.254324    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:23.254781 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.254347    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h" event={"ID":"b4c085be-1c20-4582-b15c-20d62f3884ca","Type":"ContainerStarted","Data":"a7336b6ed7c65a9170ce804aa2a059e85e965499115bc7eb1e1b634c71720727"}
Sep 09 21:11:23.272825 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.272782    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h" podStartSLOduration=1.272768092 podStartE2EDuration="1.272768092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:23.269488602 +0000 UTC m=+1371.772318629" watchObservedRunningTime="2026-09-09 21:11:23.272768092 +0000 UTC m=+1371.775598093"
Sep 09 21:11:23.285924 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.285900    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"]
Sep 09 21:11:23.289585 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.289567    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.291702 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.291678    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"openshift-service-ca.crt\""
Sep 09 21:11:23.291841 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.291811    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"default-dockercfg-bf86r\""
Sep 09 21:11:23.291932 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.291858    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"kube-root-ca.crt\""
Sep 09 21:11:23.291932 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.291923    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8db5a7\"/\"auth-test-model-8db5a7-kserve-self-signed-certs\""
Sep 09 21:11:23.300242 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.300220    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"]
Sep 09 21:11:23.407046 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.407006    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-dshm\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.407046 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.407048    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-home\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.407267 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.407073    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgdv\" (UniqueName: \"kubernetes.io/projected/931dcecf-9835-4be5-ad28-ffc113efc18f-kube-api-access-9vgdv\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.407267 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.407228    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-tmp-dir\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.407359 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.407319    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-model-cache\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.407359 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.407351    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931dcecf-9835-4be5-ad28-ffc113efc18f-tls-certs\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.508548 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508466    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-tmp-dir\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.508548 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508518    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-model-cache\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.508548 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508538    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931dcecf-9835-4be5-ad28-ffc113efc18f-tls-certs\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.508875 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508602    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-dshm\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.508875 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508763    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-home\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.508875 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508835    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgdv\" (UniqueName: \"kubernetes.io/projected/931dcecf-9835-4be5-ad28-ffc113efc18f-kube-api-access-9vgdv\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.509036 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.508971    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-model-cache\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.509205 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.509133    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-home\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.509417 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.509391    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-tmp-dir\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.510928 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.510908    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-dshm\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.511173 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.511157    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931dcecf-9835-4be5-ad28-ffc113efc18f-tls-certs\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.517085 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.517065    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgdv\" (UniqueName: \"kubernetes.io/projected/931dcecf-9835-4be5-ad28-ffc113efc18f-kube-api-access-9vgdv\") pod \"auth-test-model-8db5a7-kserve-6d874f4f6c-2z545\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") " pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.600289 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.600260    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:23.879306 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:23.879278    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"]
Sep 09 21:11:23.881053 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:11:23.881030    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod931dcecf_9835_4be5_ad28_ffc113efc18f.slice/crio-85ed70d37a9252d199c267fc8d56f3ea368048aab364ffc5b426b0e7f9dd9dff WatchSource:0}: Error finding container 85ed70d37a9252d199c267fc8d56f3ea368048aab364ffc5b426b0e7f9dd9dff: Status 404 returned error can't find the container with id 85ed70d37a9252d199c267fc8d56f3ea368048aab364ffc5b426b0e7f9dd9dff
Sep 09 21:11:24.272174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:24.272106    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545" event={"ID":"931dcecf-9835-4be5-ad28-ffc113efc18f","Type":"ContainerStarted","Data":"998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96"}
Sep 09 21:11:24.272658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:24.272187    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545" event={"ID":"931dcecf-9835-4be5-ad28-ffc113efc18f","Type":"ContainerStarted","Data":"85ed70d37a9252d199c267fc8d56f3ea368048aab364ffc5b426b0e7f9dd9dff"}
Sep 09 21:11:24.299046 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:24.298681    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545" podStartSLOduration=1.298659174 podStartE2EDuration="1.298659174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:24.297698873 +0000 UTC m=+1372.800528882" watchObservedRunningTime="2026-09-09 21:11:24.298659174 +0000 UTC m=+1372.801489176"
Sep 09 21:11:33.600823 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:33.600786    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:33.601189 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:33.600841    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:33.613994 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:33.613967    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:34.274259 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:34.274225    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:11:34.338881 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:34.338845    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:11:49.212314 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.212271    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"]
Sep 09 21:11:49.215878 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.215855    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.218260 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.218232    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f8b5626b\"/\"openshift-service-ca.crt\""
Sep 09 21:11:49.218411 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.218272    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f8b5626b\"/\"e2e-lifecycle-model-f8b5626b-kserve-self-signed-certs\""
Sep 09 21:11:49.218929 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.218909    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f8b5626b\"/\"kube-root-ca.crt\""
Sep 09 21:11:49.219024 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.219001    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f8b5626b\"/\"default-dockercfg-wh2px\""
Sep 09 21:11:49.227368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.227345    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"]
Sep 09 21:11:49.362585 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.362545    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-tmp-dir\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.362817 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.362602    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20b688f-56cf-4100-b35a-8a52454dbc06-tls-certs\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.362817 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.362663    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-dshm\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.362817 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.362765    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2lb\" (UniqueName: \"kubernetes.io/projected/f20b688f-56cf-4100-b35a-8a52454dbc06-kube-api-access-tb2lb\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.362957 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.362846    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-home\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.362957 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.362887    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-model-cache\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.463624 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.463526    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-home\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.463809 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.463713    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-model-cache\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.463872 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.463848    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-tmp-dir\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.463955 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.463924    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20b688f-56cf-4100-b35a-8a52454dbc06-tls-certs\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.464020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.463961    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-dshm\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.464020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.463993    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2lb\" (UniqueName: \"kubernetes.io/projected/f20b688f-56cf-4100-b35a-8a52454dbc06-kube-api-access-tb2lb\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.464130 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.464036    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-home\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.464130 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.464094    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-model-cache\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.464203 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.464178    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-tmp-dir\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.466225 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.466189    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-dshm\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.466448 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.466429    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20b688f-56cf-4100-b35a-8a52454dbc06-tls-certs\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.471618 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.471591    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2lb\" (UniqueName: \"kubernetes.io/projected/f20b688f-56cf-4100-b35a-8a52454dbc06-kube-api-access-tb2lb\") pod \"e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") " pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.525852 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.525819    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:49.801452 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:49.801409    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"]
Sep 09 21:11:49.803323 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:11:49.803288    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf20b688f_56cf_4100_b35a_8a52454dbc06.slice/crio-d811d44fee4104a544bd6d8a04c3963bcf60fd38fd3708be7033af1ef904758a WatchSource:0}: Error finding container d811d44fee4104a544bd6d8a04c3963bcf60fd38fd3708be7033af1ef904758a: Status 404 returned error can't find the container with id d811d44fee4104a544bd6d8a04c3963bcf60fd38fd3708be7033af1ef904758a
Sep 09 21:11:50.419681 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:50.419632    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q" event={"ID":"f20b688f-56cf-4100-b35a-8a52454dbc06","Type":"ContainerStarted","Data":"134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840"}
Sep 09 21:11:50.420123 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:50.419692    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q" event={"ID":"f20b688f-56cf-4100-b35a-8a52454dbc06","Type":"ContainerStarted","Data":"d811d44fee4104a544bd6d8a04c3963bcf60fd38fd3708be7033af1ef904758a"}
Sep 09 21:11:50.438036 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:50.437974    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q" podStartSLOduration=1.437955949 podStartE2EDuration="1.437955949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:50.435537435 +0000 UTC m=+1398.938367439" watchObservedRunningTime="2026-09-09 21:11:50.437955949 +0000 UTC m=+1398.940785948"
Sep 09 21:11:59.526383 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:59.526345    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:59.526961 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:59.526403    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:11:59.539888 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:11:59.539850    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:12:00.491868 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:00.491837    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:12:30.024783 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.024590    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"]
Sep 09 21:12:30.025253 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.024857    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h" podUID="b4c085be-1c20-4582-b15c-20d62f3884ca" containerName="payload-pre-processing" containerID="cri-o://7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3" gracePeriod=30
Sep 09 21:12:30.291905 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.291835    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:12:30.401960 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.401925    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c085be-1c20-4582-b15c-20d62f3884ca-plugins-config-volume\") pod \"b4c085be-1c20-4582-b15c-20d62f3884ca\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") "
Sep 09 21:12:30.402153 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.402071    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcbnb\" (UniqueName: \"kubernetes.io/projected/b4c085be-1c20-4582-b15c-20d62f3884ca-kube-api-access-kcbnb\") pod \"b4c085be-1c20-4582-b15c-20d62f3884ca\" (UID: \"b4c085be-1c20-4582-b15c-20d62f3884ca\") "
Sep 09 21:12:30.402278 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.402251    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c085be-1c20-4582-b15c-20d62f3884ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4c085be-1c20-4582-b15c-20d62f3884ca" (UID: "b4c085be-1c20-4582-b15c-20d62f3884ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:30.402427 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.402411    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4c085be-1c20-4582-b15c-20d62f3884ca-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:30.404091 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.404069    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c085be-1c20-4582-b15c-20d62f3884ca-kube-api-access-kcbnb" (OuterVolumeSpecName: "kube-api-access-kcbnb") pod "b4c085be-1c20-4582-b15c-20d62f3884ca" (UID: "b4c085be-1c20-4582-b15c-20d62f3884ca"). InnerVolumeSpecName "kube-api-access-kcbnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:30.503687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.503661    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcbnb\" (UniqueName: \"kubernetes.io/projected/b4c085be-1c20-4582-b15c-20d62f3884ca-kube-api-access-kcbnb\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:30.647179 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.647093    2548 generic.go:356] "Generic (PLEG): container finished" podID="b4c085be-1c20-4582-b15c-20d62f3884ca" containerID="7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3" exitCode=0
Sep 09 21:12:30.647179 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.647144    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"
Sep 09 21:12:30.647487 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.647174    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h" event={"ID":"b4c085be-1c20-4582-b15c-20d62f3884ca","Type":"ContainerDied","Data":"7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3"}
Sep 09 21:12:30.647487 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.647232    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h" event={"ID":"b4c085be-1c20-4582-b15c-20d62f3884ca","Type":"ContainerDied","Data":"a7336b6ed7c65a9170ce804aa2a059e85e965499115bc7eb1e1b634c71720727"}
Sep 09 21:12:30.647487 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.647256    2548 scope.go:117] "RemoveContainer" containerID="7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3"
Sep 09 21:12:30.660387 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.660302    2548 scope.go:117] "RemoveContainer" containerID="7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3"
Sep 09 21:12:30.660806 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:12:30.660777    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3\": container with ID starting with 7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3 not found: ID does not exist" containerID="7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3"
Sep 09 21:12:30.660923 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.660813    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3"} err="failed to get container status \"7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3\": rpc error: code = NotFound desc = could not find container \"7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3\": container with ID starting with 7f5f05d7653595c6eafe385d3b164a09dbc8a16177fc230ace2289597011ffa3 not found: ID does not exist"
Sep 09 21:12:30.672203 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.672175    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"]
Sep 09 21:12:30.675101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:30.675079    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8b5626b-c86fffb6-t7z4h"]
Sep 09 21:12:31.970888 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:31.970854    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c085be-1c20-4582-b15c-20d62f3884ca" path="/var/lib/kubelet/pods/b4c085be-1c20-4582-b15c-20d62f3884ca/volumes"
Sep 09 21:12:38.295480 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:38.295444    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"]
Sep 09 21:12:53.315556 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.315488    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q" podUID="f20b688f-56cf-4100-b35a-8a52454dbc06" containerName="main" containerID="cri-o://134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840" gracePeriod=45
Sep 09 21:12:53.582942 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.582908    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:12:53.633103 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633071    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-home\") pod \"f20b688f-56cf-4100-b35a-8a52454dbc06\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") "
Sep 09 21:12:53.633300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633115    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20b688f-56cf-4100-b35a-8a52454dbc06-tls-certs\") pod \"f20b688f-56cf-4100-b35a-8a52454dbc06\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") "
Sep 09 21:12:53.633300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633162    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-dshm\") pod \"f20b688f-56cf-4100-b35a-8a52454dbc06\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") "
Sep 09 21:12:53.633300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633208    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-tmp-dir\") pod \"f20b688f-56cf-4100-b35a-8a52454dbc06\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") "
Sep 09 21:12:53.633300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633262    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-model-cache\") pod \"f20b688f-56cf-4100-b35a-8a52454dbc06\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") "
Sep 09 21:12:53.633514 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633302    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb2lb\" (UniqueName: \"kubernetes.io/projected/f20b688f-56cf-4100-b35a-8a52454dbc06-kube-api-access-tb2lb\") pod \"f20b688f-56cf-4100-b35a-8a52454dbc06\" (UID: \"f20b688f-56cf-4100-b35a-8a52454dbc06\") "
Sep 09 21:12:53.633514 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633421    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-home" (OuterVolumeSpecName: "home") pod "f20b688f-56cf-4100-b35a-8a52454dbc06" (UID: "f20b688f-56cf-4100-b35a-8a52454dbc06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:53.633683 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633656    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-model-cache" (OuterVolumeSpecName: "model-cache") pod "f20b688f-56cf-4100-b35a-8a52454dbc06" (UID: "f20b688f-56cf-4100-b35a-8a52454dbc06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:53.633766 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633720    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f20b688f-56cf-4100-b35a-8a52454dbc06" (UID: "f20b688f-56cf-4100-b35a-8a52454dbc06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:53.633830 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.633774    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-home\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:53.635833 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.635792    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20b688f-56cf-4100-b35a-8a52454dbc06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f20b688f-56cf-4100-b35a-8a52454dbc06" (UID: "f20b688f-56cf-4100-b35a-8a52454dbc06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:53.636053 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.636030    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-dshm" (OuterVolumeSpecName: "dshm") pod "f20b688f-56cf-4100-b35a-8a52454dbc06" (UID: "f20b688f-56cf-4100-b35a-8a52454dbc06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:53.636113 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.636030    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20b688f-56cf-4100-b35a-8a52454dbc06-kube-api-access-tb2lb" (OuterVolumeSpecName: "kube-api-access-tb2lb") pod "f20b688f-56cf-4100-b35a-8a52454dbc06" (UID: "f20b688f-56cf-4100-b35a-8a52454dbc06"). InnerVolumeSpecName "kube-api-access-tb2lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:53.735039 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.734995    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-tmp-dir\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:53.735039 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.735031    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-model-cache\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:53.735039 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.735043    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2lb\" (UniqueName: \"kubernetes.io/projected/f20b688f-56cf-4100-b35a-8a52454dbc06-kube-api-access-tb2lb\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:53.735284 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.735054    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20b688f-56cf-4100-b35a-8a52454dbc06-tls-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:53.735284 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.735063    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20b688f-56cf-4100-b35a-8a52454dbc06-dshm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:12:53.771093 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.771058    2548 generic.go:356] "Generic (PLEG): container finished" podID="f20b688f-56cf-4100-b35a-8a52454dbc06" containerID="134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840" exitCode=0
Sep 09 21:12:53.771248 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.771099    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"
Sep 09 21:12:53.771248 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.771144    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q" event={"ID":"f20b688f-56cf-4100-b35a-8a52454dbc06","Type":"ContainerDied","Data":"134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840"}
Sep 09 21:12:53.771248 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.771196    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q" event={"ID":"f20b688f-56cf-4100-b35a-8a52454dbc06","Type":"ContainerDied","Data":"d811d44fee4104a544bd6d8a04c3963bcf60fd38fd3708be7033af1ef904758a"}
Sep 09 21:12:53.771248 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.771217    2548 scope.go:117] "RemoveContainer" containerID="134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840"
Sep 09 21:12:53.781004 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.780986    2548 scope.go:117] "RemoveContainer" containerID="134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840"
Sep 09 21:12:53.781280 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:12:53.781261    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840\": container with ID starting with 134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840 not found: ID does not exist" containerID="134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840"
Sep 09 21:12:53.781320 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.781289    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840"} err="failed to get container status \"134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840\": rpc error: code = NotFound desc = could not find container \"134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840\": container with ID starting with 134a843efbcec7d067fde767121481b822b9f60d366383a7187e27c05a3c3840 not found: ID does not exist"
Sep 09 21:12:53.792987 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.792961    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"]
Sep 09 21:12:53.798029 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.798007    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f8b5626b/e2e-lifecycle-model-f8b5626b-kserve-66d7f5cf9-24p6q"]
Sep 09 21:12:53.969619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:12:53.969577    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20b688f-56cf-4100-b35a-8a52454dbc06" path="/var/lib/kubelet/pods/f20b688f-56cf-4100-b35a-8a52454dbc06/volumes"
Sep 09 21:13:32.043514 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:32.043477    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:13:32.049080 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:32.049056    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:13:37.100066 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:37.100025    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"]
Sep 09 21:13:52.119792 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.119721    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545" podUID="931dcecf-9835-4be5-ad28-ffc113efc18f" containerName="main" containerID="cri-o://998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96" gracePeriod=45
Sep 09 21:13:52.386414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.386352    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:13:52.420110 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420073    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931dcecf-9835-4be5-ad28-ffc113efc18f-tls-certs\") pod \"931dcecf-9835-4be5-ad28-ffc113efc18f\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") "
Sep 09 21:13:52.420250 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420168    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vgdv\" (UniqueName: \"kubernetes.io/projected/931dcecf-9835-4be5-ad28-ffc113efc18f-kube-api-access-9vgdv\") pod \"931dcecf-9835-4be5-ad28-ffc113efc18f\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") "
Sep 09 21:13:52.420250 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420210    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-model-cache\") pod \"931dcecf-9835-4be5-ad28-ffc113efc18f\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") "
Sep 09 21:13:52.420250 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420228    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-dshm\") pod \"931dcecf-9835-4be5-ad28-ffc113efc18f\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") "
Sep 09 21:13:52.420380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420274    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-home\") pod \"931dcecf-9835-4be5-ad28-ffc113efc18f\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") "
Sep 09 21:13:52.420762 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420440    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-model-cache" (OuterVolumeSpecName: "model-cache") pod "931dcecf-9835-4be5-ad28-ffc113efc18f" (UID: "931dcecf-9835-4be5-ad28-ffc113efc18f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:52.420762 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420676    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-model-cache\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:13:52.420762 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.420688    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-home" (OuterVolumeSpecName: "home") pod "931dcecf-9835-4be5-ad28-ffc113efc18f" (UID: "931dcecf-9835-4be5-ad28-ffc113efc18f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:52.422379 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.422350    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-dshm" (OuterVolumeSpecName: "dshm") pod "931dcecf-9835-4be5-ad28-ffc113efc18f" (UID: "931dcecf-9835-4be5-ad28-ffc113efc18f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:52.422560 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.422415    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931dcecf-9835-4be5-ad28-ffc113efc18f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "931dcecf-9835-4be5-ad28-ffc113efc18f" (UID: "931dcecf-9835-4be5-ad28-ffc113efc18f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:13:52.422657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.422631    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931dcecf-9835-4be5-ad28-ffc113efc18f-kube-api-access-9vgdv" (OuterVolumeSpecName: "kube-api-access-9vgdv") pod "931dcecf-9835-4be5-ad28-ffc113efc18f" (UID: "931dcecf-9835-4be5-ad28-ffc113efc18f"). InnerVolumeSpecName "kube-api-access-9vgdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:13:52.521882 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.521830    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-tmp-dir\") pod \"931dcecf-9835-4be5-ad28-ffc113efc18f\" (UID: \"931dcecf-9835-4be5-ad28-ffc113efc18f\") "
Sep 09 21:13:52.522162 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.522137    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "931dcecf-9835-4be5-ad28-ffc113efc18f" (UID: "931dcecf-9835-4be5-ad28-ffc113efc18f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:13:52.522236 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.522220    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgdv\" (UniqueName: \"kubernetes.io/projected/931dcecf-9835-4be5-ad28-ffc113efc18f-kube-api-access-9vgdv\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:13:52.522270 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.522240    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-dshm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:13:52.522270 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.522251    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-home\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:13:52.522339 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.522275    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931dcecf-9835-4be5-ad28-ffc113efc18f-tmp-dir\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:13:52.522339 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:52.522283    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931dcecf-9835-4be5-ad28-ffc113efc18f-tls-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:13:53.117474 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.117438    2548 generic.go:356] "Generic (PLEG): container finished" podID="931dcecf-9835-4be5-ad28-ffc113efc18f" containerID="998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96" exitCode=0
Sep 09 21:13:53.117658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.117484    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"
Sep 09 21:13:53.117658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.117480    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545" event={"ID":"931dcecf-9835-4be5-ad28-ffc113efc18f","Type":"ContainerDied","Data":"998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96"}
Sep 09 21:13:53.117658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.117594    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545" event={"ID":"931dcecf-9835-4be5-ad28-ffc113efc18f","Type":"ContainerDied","Data":"85ed70d37a9252d199c267fc8d56f3ea368048aab364ffc5b426b0e7f9dd9dff"}
Sep 09 21:13:53.117658 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.117616    2548 scope.go:117] "RemoveContainer" containerID="998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96"
Sep 09 21:13:53.127404 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.127224    2548 scope.go:117] "RemoveContainer" containerID="998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96"
Sep 09 21:13:53.127691 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:13:53.127517    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96\": container with ID starting with 998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96 not found: ID does not exist" containerID="998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96"
Sep 09 21:13:53.127691 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.127545    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96"} err="failed to get container status \"998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96\": rpc error: code = NotFound desc = could not find container \"998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96\": container with ID starting with 998d95e3ece49f84cb1366793f892697b08509a2a3a77f6b37b7bbe82f4afa96 not found: ID does not exist"
Sep 09 21:13:53.138175 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.138137    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"]
Sep 09 21:13:53.142305 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.142284    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8db5a7/auth-test-model-8db5a7-kserve-6d874f4f6c-2z545"]
Sep 09 21:13:53.969417 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:13:53.969382    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931dcecf-9835-4be5-ad28-ffc113efc18f" path="/var/lib/kubelet/pods/931dcecf-9835-4be5-ad28-ffc113efc18f/volumes"
Sep 09 21:14:17.136013 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.135975    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"]
Sep 09 21:14:17.136512 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136463    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931dcecf-9835-4be5-ad28-ffc113efc18f" containerName="main"
Sep 09 21:14:17.136512 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136482    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="931dcecf-9835-4be5-ad28-ffc113efc18f" containerName="main"
Sep 09 21:14:17.136512 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136501    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20b688f-56cf-4100-b35a-8a52454dbc06" containerName="main"
Sep 09 21:14:17.136512 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136508    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20b688f-56cf-4100-b35a-8a52454dbc06" containerName="main"
Sep 09 21:14:17.136778 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136532    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c085be-1c20-4582-b15c-20d62f3884ca" containerName="payload-pre-processing"
Sep 09 21:14:17.136778 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136539    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c085be-1c20-4582-b15c-20d62f3884ca" containerName="payload-pre-processing"
Sep 09 21:14:17.136778 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136619    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c085be-1c20-4582-b15c-20d62f3884ca" containerName="payload-pre-processing"
Sep 09 21:14:17.136778 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136634    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="931dcecf-9835-4be5-ad28-ffc113efc18f" containerName="main"
Sep 09 21:14:17.136778 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.136644    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20b688f-56cf-4100-b35a-8a52454dbc06" containerName="main"
Sep 09 21:14:17.139304 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.139284    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.143381 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.143361    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dc3099-openshift-default-dockercfg-lvhzd\""
Sep 09 21:14:17.159542 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.159517    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"]
Sep 09 21:14:17.266575 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266530    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-token\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266776 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266594    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istiod-ca-cert\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266776 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266631    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvfm7\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-kube-api-access-mvfm7\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266776 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266660    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-credential-socket\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266776 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266686    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-data\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266776 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266721    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-podinfo\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266998 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266783    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-certs\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266998 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266844    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-socket\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.266998 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.266865    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-envoy\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.367966    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-credential-socket\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368023 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368026    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-data\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368271 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368069    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-podinfo\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368271 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368110    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-certs\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368271 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368180    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-socket\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368271 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368218    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-envoy\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368275    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-token\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368323    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istiod-ca-cert\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368368    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvfm7\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-kube-api-access-mvfm7\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368447    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-credential-socket\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368475    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-data\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368745 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368557    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-socket\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.368745 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.368660    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-certs\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.369079 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.369061    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istiod-ca-cert\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.370575 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.370549    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-podinfo\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.370575 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.370560    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-envoy\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.379317 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.379264    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvfm7\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-kube-api-access-mvfm7\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.380470 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.380444    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-token\") pod \"e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") " pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.450186 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.450154    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:17.733754 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.733706    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"]
Sep 09 21:14:17.735985 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:14:17.735954    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1963e4e0_e07c_4a6b_9f4c_10ca91670c65.slice/crio-3d2adfb1ef834acd165fececfbb7f945a3aa10d8508619f098020897879be215 WatchSource:0}: Error finding container 3d2adfb1ef834acd165fececfbb7f945a3aa10d8508619f098020897879be215: Status 404 returned error can't find the container with id 3d2adfb1ef834acd165fececfbb7f945a3aa10d8508619f098020897879be215
Sep 09 21:14:17.738484 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.738452    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:17.738591 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.738520    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:17.738591 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:17.738554    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:18.254468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:18.254423    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" event={"ID":"1963e4e0-e07c-4a6b-9f4c-10ca91670c65","Type":"ContainerStarted","Data":"04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"}
Sep 09 21:14:18.254972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:18.254498    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" event={"ID":"1963e4e0-e07c-4a6b-9f4c-10ca91670c65","Type":"ContainerStarted","Data":"3d2adfb1ef834acd165fececfbb7f945a3aa10d8508619f098020897879be215"}
Sep 09 21:14:18.277625 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:18.277570    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podStartSLOduration=1.277553146 podStartE2EDuration="1.277553146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:18.276889934 +0000 UTC m=+1546.779719936" watchObservedRunningTime="2026-09-09 21:14:18.277553146 +0000 UTC m=+1546.780383147"
Sep 09 21:14:18.450561 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:18.450525    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:18.456704 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:18.456674    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:19.259748 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:19.259711    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:19.260790 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:19.260756    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:41.386015 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:41.385929    2548 generic.go:356] "Generic (PLEG): container finished" podID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerID="04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84" exitCode=0
Sep 09 21:14:41.386401 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:41.386011    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" event={"ID":"1963e4e0-e07c-4a6b-9f4c-10ca91670c65","Type":"ContainerDied","Data":"04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"}
Sep 09 21:14:41.386541 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:41.386521    2548 scope.go:117] "RemoveContainer" containerID="04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"
Sep 09 21:14:41.387710 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:41.387683    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:41.387783 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:41.387769    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:41.387821 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:41.387805    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:42.393251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:42.393217    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" event={"ID":"1963e4e0-e07c-4a6b-9f4c-10ca91670c65","Type":"ContainerStarted","Data":"f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf"}
Sep 09 21:14:42.466987 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:42.466937    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:42.467310 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:42.467282    2548 patch_prober.go:28] interesting pod/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 21:14:42.467485 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:42.467461    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 21:14:43.451276 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:43.451232    2548 patch_prober.go:28] interesting pod/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 21:14:43.451908 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:43.451868    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 21:14:44.450881 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:44.450842    2548 patch_prober.go:28] interesting pod/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 21:14:44.451181 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:44.451152    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 21:14:45.451143 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.451102    2548 patch_prober.go:28] interesting pod/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 21:14:45.451709 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.451673    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 21:14:45.876810 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.876688    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"]
Sep 09 21:14:45.880366 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.880341    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.883821 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.883799    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bd9741-openshift-default-dockercfg-27tf6\""
Sep 09 21:14:45.892169 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.892144    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"]
Sep 09 21:14:45.980396 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.980355    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrfs\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-kube-api-access-mrrfs\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.980653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.980632    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-socket\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.980884 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.980864    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-podinfo\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.981018 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.981003    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istiod-ca-cert\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.981144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.981127    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-data\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.981276 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.981262    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-token\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.981390 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.981374    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-certs\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.981526 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.981510    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-credential-socket\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:45.981673 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:45.981657    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-envoy\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083005 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.082967    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrfs\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-kube-api-access-mrrfs\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083018    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-socket\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083053    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-podinfo\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083088    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istiod-ca-cert\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083126    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-data\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083345 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083174    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-token\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083345 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083200    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-certs\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083345 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083240    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-credential-socket\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083296    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-envoy\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083592 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083517    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-socket\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083592 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083547    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-data\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083717 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083695    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-credential-socket\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083797 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083772    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-certs\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.083937 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.083909    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istiod-ca-cert\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.085565 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.085543    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-envoy\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.085675 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.085607    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-podinfo\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.090893 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.090868    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrfs\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-kube-api-access-mrrfs\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.091015 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.090994    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-token\") pod \"e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") " pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.193144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.193116    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:46.455241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.455144    2548 patch_prober.go:28] interesting pod/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 21:14:46.455241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.455212    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 21:14:46.479199 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.479173    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"]
Sep 09 21:14:46.481216 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:14:46.481185    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2954e2ee_da33_4e9f_a13b_c19a14822b0e.slice/crio-26faf946185199898cf5e010578bb7205a8d2db4362963af0df031924571c844 WatchSource:0}: Error finding container 26faf946185199898cf5e010578bb7205a8d2db4362963af0df031924571c844: Status 404 returned error can't find the container with id 26faf946185199898cf5e010578bb7205a8d2db4362963af0df031924571c844
Sep 09 21:14:46.483468 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.483428    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:46.483582 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.483513    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:46.483582 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:46.483557    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:14:47.420950 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:47.420910    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" event={"ID":"2954e2ee-da33-4e9f-a13b-c19a14822b0e","Type":"ContainerStarted","Data":"0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"}
Sep 09 21:14:47.421159 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:47.420973    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" event={"ID":"2954e2ee-da33-4e9f-a13b-c19a14822b0e","Type":"ContainerStarted","Data":"26faf946185199898cf5e010578bb7205a8d2db4362963af0df031924571c844"}
Sep 09 21:14:47.449587 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:47.448798    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podStartSLOduration=2.448774486 podStartE2EDuration="2.448774486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:47.445166815 +0000 UTC m=+1575.947996847" watchObservedRunningTime="2026-09-09 21:14:47.448774486 +0000 UTC m=+1575.951604487"
Sep 09 21:14:47.451479 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:47.451447    2548 patch_prober.go:28] interesting pod/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 09 21:14:47.451635 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:47.451508    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 09 21:14:48.194336 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.194295    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:48.199753 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.199700    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:48.427077 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.427041    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:48.427992 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.427964    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:14:48.457149 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.457067    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:48.457493 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.457469    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:48.458328 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:48.458301    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:14:51.394417 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.394372    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"]
Sep 09 21:14:51.399294 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.399270    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.404072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.404051    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bd9741-dockercfg-2qbrm\""
Sep 09 21:14:51.404578 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.404561    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bd9741\""
Sep 09 21:14:51.430326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.430295    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"]
Sep 09 21:14:51.435326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.435296    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569bae39-0f41-477e-955e-7b2248e16073-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.435460 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.435433    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffq4\" (UniqueName: \"kubernetes.io/projected/569bae39-0f41-477e-955e-7b2248e16073-kube-api-access-nffq4\") pod \"payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.537115 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.537069    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569bae39-0f41-477e-955e-7b2248e16073-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.537346 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.537206    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffq4\" (UniqueName: \"kubernetes.io/projected/569bae39-0f41-477e-955e-7b2248e16073-kube-api-access-nffq4\") pod \"payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.537881 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.537853    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569bae39-0f41-477e-955e-7b2248e16073-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.549170 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.549129    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffq4\" (UniqueName: \"kubernetes.io/projected/569bae39-0f41-477e-955e-7b2248e16073-kube-api-access-nffq4\") pod \"payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.711267 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.711234    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:51.990443 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:51.990417    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"]
Sep 09 21:14:51.992909 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:14:51.992880    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569bae39_0f41_477e_955e_7b2248e16073.slice/crio-5d2a7c8ba973282b24d2daf09605ed1b7126ece648f1251e5e69299362fa2726 WatchSource:0}: Error finding container 5d2a7c8ba973282b24d2daf09605ed1b7126ece648f1251e5e69299362fa2726: Status 404 returned error can't find the container with id 5d2a7c8ba973282b24d2daf09605ed1b7126ece648f1251e5e69299362fa2726
Sep 09 21:14:52.451853 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:52.451814    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm" event={"ID":"569bae39-0f41-477e-955e-7b2248e16073","Type":"ContainerStarted","Data":"247a9eebb5ad0d2f117e039d2a7841671560de75124cabe5033246f28129dcb6"}
Sep 09 21:14:52.452247 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:52.451877    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm" event={"ID":"569bae39-0f41-477e-955e-7b2248e16073","Type":"ContainerStarted","Data":"5d2a7c8ba973282b24d2daf09605ed1b7126ece648f1251e5e69299362fa2726"}
Sep 09 21:14:52.452247 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:52.451972    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:14:52.470211 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:14:52.470159    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm" podStartSLOduration=1.470143 podStartE2EDuration="1.470143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:52.468420318 +0000 UTC m=+1580.971250320" watchObservedRunningTime="2026-09-09 21:14:52.470143 +0000 UTC m=+1580.972973000"
Sep 09 21:15:00.154018 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:00.153975    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"]
Sep 09 21:15:00.159285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:00.159253    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"
Sep 09 21:15:00.163956 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:00.163918    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6l2wq\""
Sep 09 21:15:00.174492 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:00.174468    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"
Sep 09 21:15:00.185704 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:00.184608    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"]
Sep 09 21:15:00.510412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:00.510385    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"]
Sep 09 21:15:00.511937 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:15:00.511897    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d87220b_db55_4662_9dad_ac65fe30643f.slice/crio-31443255b6c6e3de0b441d224e3b15ac4892bad4643d2309a523f1905be53152 WatchSource:0}: Error finding container 31443255b6c6e3de0b441d224e3b15ac4892bad4643d2309a523f1905be53152: Status 404 returned error can't find the container with id 31443255b6c6e3de0b441d224e3b15ac4892bad4643d2309a523f1905be53152
Sep 09 21:15:01.503434 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:01.503385    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr" event={"ID":"6d87220b-db55-4662-9dad-ac65fe30643f","Type":"ContainerStarted","Data":"31443255b6c6e3de0b441d224e3b15ac4892bad4643d2309a523f1905be53152"}
Sep 09 21:15:03.459467 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:03.459435    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:15:03.518679 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:03.518634    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr" event={"ID":"6d87220b-db55-4662-9dad-ac65fe30643f","Type":"ContainerStarted","Data":"c24fd1a6090de89c6acb079fe8eee4accf68a0e12d180b9502d71d54597b7303"}
Sep 09 21:15:04.416220 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.416107    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr" podStartSLOduration=1.833870991 podStartE2EDuration="4.416086214s" totalImagesPullingTime="2.582215223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:03.534795825 +0000 UTC m=+1592.037625827" watchObservedRunningTime="2026-09-09 21:15:04.416086214 +0000 UTC m=+1592.918916215"
Sep 09 21:15:04.418310 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.418287    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"]
Sep 09 21:15:04.421878 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.421858    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.425003 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.424979    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"kube-root-ca.crt\""
Sep 09 21:15:04.425676 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.425649    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"openshift-service-ca.crt\""
Sep 09 21:15:04.425827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.425649    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"default-dockercfg-vb4x7\""
Sep 09 21:15:04.425827 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.425721    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3d3cac\"/\"rate-test-model-3d3cac-kserve-self-signed-certs\""
Sep 09 21:15:04.435750 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.435704    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"]
Sep 09 21:15:04.469952 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.469909    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:15:04.470435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.469996    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:15:04.525973 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.525939    2548 generic.go:356] "Generic (PLEG): container finished" podID="6d87220b-db55-4662-9dad-ac65fe30643f" containerID="c24fd1a6090de89c6acb079fe8eee4accf68a0e12d180b9502d71d54597b7303" exitCode=0
Sep 09 21:15:04.526135 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.525975    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr" event={"ID":"6d87220b-db55-4662-9dad-ac65fe30643f","Type":"ContainerDied","Data":"c24fd1a6090de89c6acb079fe8eee4accf68a0e12d180b9502d71d54597b7303"}
Sep 09 21:15:04.604241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.604200    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-dshm\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.604241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.604252    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-home\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.604515 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.604336    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0b983b-4880-4b1a-80bb-293e99cc7277-tls-certs\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.604515 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.604404    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-tmp-dir\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.604515 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.604447    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8bf6\" (UniqueName: \"kubernetes.io/projected/3c0b983b-4880-4b1a-80bb-293e99cc7277-kube-api-access-l8bf6\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.604515 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.604496    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-model-cache\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706037 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.705988    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-dshm\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706218 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706048    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-home\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706218 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706121    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0b983b-4880-4b1a-80bb-293e99cc7277-tls-certs\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706218 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706184    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-tmp-dir\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706390 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706222    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8bf6\" (UniqueName: \"kubernetes.io/projected/3c0b983b-4880-4b1a-80bb-293e99cc7277-kube-api-access-l8bf6\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706390 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706267    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-model-cache\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706584 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706556    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-home\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706662 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706570    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-tmp-dir\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.706662 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.706607    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-model-cache\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.708498 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.708465    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-dshm\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.708966 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.708942    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0b983b-4880-4b1a-80bb-293e99cc7277-tls-certs\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.714926 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.714900    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8bf6\" (UniqueName: \"kubernetes.io/projected/3c0b983b-4880-4b1a-80bb-293e99cc7277-kube-api-access-l8bf6\") pod \"rate-test-model-3d3cac-kserve-8594f4658c-rg4r7\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") " pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:04.731979 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:04.731950    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:05.021763 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:05.021719    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"]
Sep 09 21:15:05.024207 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:15:05.024179    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c0b983b_4880_4b1a_80bb_293e99cc7277.slice/crio-cf1fec7ecb422c227714eb21aa50be5a8183cf5b0225ce67a9300f7c8939a18b WatchSource:0}: Error finding container cf1fec7ecb422c227714eb21aa50be5a8183cf5b0225ce67a9300f7c8939a18b: Status 404 returned error can't find the container with id cf1fec7ecb422c227714eb21aa50be5a8183cf5b0225ce67a9300f7c8939a18b
Sep 09 21:15:05.533155 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:05.533120    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7" event={"ID":"3c0b983b-4880-4b1a-80bb-293e99cc7277","Type":"ContainerStarted","Data":"f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec"}
Sep 09 21:15:05.533645 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:05.533171    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7" event={"ID":"3c0b983b-4880-4b1a-80bb-293e99cc7277","Type":"ContainerStarted","Data":"cf1fec7ecb422c227714eb21aa50be5a8183cf5b0225ce67a9300f7c8939a18b"}
Sep 09 21:15:05.696986 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:05.696951    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"
Sep 09 21:15:05.714430 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:05.714364    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7" podStartSLOduration=1.7143453229999999 podStartE2EDuration="1.714345323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:05.55296751 +0000 UTC m=+1594.055797507" watchObservedRunningTime="2026-09-09 21:15:05.714345323 +0000 UTC m=+1594.217175324"
Sep 09 21:15:06.540126 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:06.540095    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"
Sep 09 21:15:06.540605 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:06.540135    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr" event={"ID":"6d87220b-db55-4662-9dad-ac65fe30643f","Type":"ContainerDied","Data":"31443255b6c6e3de0b441d224e3b15ac4892bad4643d2309a523f1905be53152"}
Sep 09 21:15:06.540605 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:06.540187    2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31443255b6c6e3de0b441d224e3b15ac4892bad4643d2309a523f1905be53152"
Sep 09 21:15:07.548373 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:07.548335    2548 generic.go:356] "Generic (PLEG): container finished" podID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerID="0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709" exitCode=0
Sep 09 21:15:07.548932 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:07.548408    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" event={"ID":"2954e2ee-da33-4e9f-a13b-c19a14822b0e","Type":"ContainerDied","Data":"0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"}
Sep 09 21:15:07.549008 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:07.548980    2548 scope.go:117] "RemoveContainer" containerID="0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"
Sep 09 21:15:07.550390 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:07.550361    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:15:07.550516 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:07.550431    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:15:07.550516 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:07.550462    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:15:08.556462 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:08.556423    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" event={"ID":"2954e2ee-da33-4e9f-a13b-c19a14822b0e","Type":"ContainerStarted","Data":"4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079"}
Sep 09 21:15:09.193988 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:09.193947    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:15:09.194254 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:09.194226    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 21:15:09.194338 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:09.194310    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 21:15:10.193758 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:10.193608    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 21:15:10.193758 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:10.193671    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 21:15:11.193621 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:11.193574    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 21:15:11.193859 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:11.193638    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 21:15:12.193834 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:12.193798    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 21:15:12.194025 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:12.193854    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 21:15:13.194238 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:13.194200    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 21:15:13.194647 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:13.194265    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 21:15:14.194088 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:14.194048    2548 patch_prober.go:28] interesting pod/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 09 21:15:14.194280 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:14.194109    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 09 21:15:14.732242 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:14.732206    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:14.732422 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:14.732257    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:14.745233 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:14.745200    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:15.198557 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:15.198526    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:15:15.198971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:15.198939    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:15:15.199587 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:15.199567    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:15:15.617135 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:15.617049    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:15:39.681901 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:39.681812    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"]
Sep 09 21:15:39.682453 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:39.682082    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy" containerID="cri-o://f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf" gracePeriod=30
Sep 09 21:15:44.948844 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:44.948768    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:15:45.132502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132469    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-podinfo\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132711 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132516    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-data\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132711 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132560    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-credential-socket\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132711 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132596    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istiod-ca-cert\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132711 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132642    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvfm7\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-kube-api-access-mvfm7\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132979 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132839    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-certs\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132979 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132871    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-data" (OuterVolumeSpecName: "istio-data") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:45.132979 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132887    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-envoy\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.132979 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.132959    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:45.133246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133000    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-token\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.133246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133072    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-socket\") pod \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\" (UID: \"1963e4e0-e07c-4a6b-9f4c-10ca91670c65\") "
Sep 09 21:15:45.133246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133087    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:45.133403 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133257    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:45.133403 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133359    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:45.133625 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133605    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.133690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133626    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.133690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133635    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.133690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133645    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.133690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.133653    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.134949 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.134920    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:45.135071 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.135017    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:15:45.135269 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.135247    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-kube-api-access-mvfm7" (OuterVolumeSpecName: "kube-api-access-mvfm7") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "kube-api-access-mvfm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:45.135313 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.135274    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-token" (OuterVolumeSpecName: "istio-token") pod "1963e4e0-e07c-4a6b-9f4c-10ca91670c65" (UID: "1963e4e0-e07c-4a6b-9f4c-10ca91670c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:45.234720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.234641    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.234720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.234670    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvfm7\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-kube-api-access-mvfm7\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.234720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.234681    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.234720 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.234689    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1963e4e0-e07c-4a6b-9f4c-10ca91670c65-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:45.780304 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.780272    2548 generic.go:356] "Generic (PLEG): container finished" podID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerID="f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf" exitCode=0
Sep 09 21:15:45.780490 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.780376    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"
Sep 09 21:15:45.780490 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.780368    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" event={"ID":"1963e4e0-e07c-4a6b-9f4c-10ca91670c65","Type":"ContainerDied","Data":"f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf"}
Sep 09 21:15:45.780572 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.780528    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9" event={"ID":"1963e4e0-e07c-4a6b-9f4c-10ca91670c65","Type":"ContainerDied","Data":"3d2adfb1ef834acd165fececfbb7f945a3aa10d8508619f098020897879be215"}
Sep 09 21:15:45.780572 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.780559    2548 scope.go:117] "RemoveContainer" containerID="f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf"
Sep 09 21:15:45.790336 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.790320    2548 scope.go:117] "RemoveContainer" containerID="04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"
Sep 09 21:15:45.801026 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.801001    2548 scope.go:117] "RemoveContainer" containerID="f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf"
Sep 09 21:15:45.801450 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:15:45.801421    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf\": container with ID starting with f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf not found: ID does not exist" containerID="f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf"
Sep 09 21:15:45.801565 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.801465    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf"} err="failed to get container status \"f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf\": rpc error: code = NotFound desc = could not find container \"f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf\": container with ID starting with f42d05bf91b74598cf24b4d1d8608c8764465d6d87f07ca4f270165f14fe3caf not found: ID does not exist"
Sep 09 21:15:45.801565 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.801489    2548 scope.go:117] "RemoveContainer" containerID="04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"
Sep 09 21:15:45.802213 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:15:45.802189    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84\": container with ID starting with 04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84 not found: ID does not exist" containerID="04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"
Sep 09 21:15:45.802316 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.802217    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84"} err="failed to get container status \"04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84\": rpc error: code = NotFound desc = could not find container \"04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84\": container with ID starting with 04d831f73459368e2d985042c6af9b1e9e865430fb1096cb7ce7e9c4a53e7a84 not found: ID does not exist"
Sep 09 21:15:45.804065 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.804044    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"]
Sep 09 21:15:45.807430 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.807412    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dc3099-openshift-default-665cb6c96d-d9zs9"]
Sep 09 21:15:45.969904 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:45.969870    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" path="/var/lib/kubelet/pods/1963e4e0-e07c-4a6b-9f4c-10ca91670c65/volumes"
Sep 09 21:15:55.660413 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:55.660375    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"]
Sep 09 21:15:55.660922 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:55.660690    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm" podUID="569bae39-0f41-477e-955e-7b2248e16073" containerName="payload-pre-processing" containerID="cri-o://247a9eebb5ad0d2f117e039d2a7841671560de75124cabe5033246f28129dcb6" gracePeriod=30
Sep 09 21:15:55.840411 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:55.840378    2548 generic.go:356] "Generic (PLEG): container finished" podID="569bae39-0f41-477e-955e-7b2248e16073" containerID="247a9eebb5ad0d2f117e039d2a7841671560de75124cabe5033246f28129dcb6" exitCode=0
Sep 09 21:15:55.840584 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:55.840446    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm" event={"ID":"569bae39-0f41-477e-955e-7b2248e16073","Type":"ContainerDied","Data":"247a9eebb5ad0d2f117e039d2a7841671560de75124cabe5033246f28129dcb6"}
Sep 09 21:15:55.919853 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:55.919787    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:15:56.044721 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.044685    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569bae39-0f41-477e-955e-7b2248e16073-plugins-config-volume\") pod \"569bae39-0f41-477e-955e-7b2248e16073\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") "
Sep 09 21:15:56.044964 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.044794    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nffq4\" (UniqueName: \"kubernetes.io/projected/569bae39-0f41-477e-955e-7b2248e16073-kube-api-access-nffq4\") pod \"569bae39-0f41-477e-955e-7b2248e16073\" (UID: \"569bae39-0f41-477e-955e-7b2248e16073\") "
Sep 09 21:15:56.045511 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.045482    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569bae39-0f41-477e-955e-7b2248e16073-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "569bae39-0f41-477e-955e-7b2248e16073" (UID: "569bae39-0f41-477e-955e-7b2248e16073"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:56.047352 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.047322    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569bae39-0f41-477e-955e-7b2248e16073-kube-api-access-nffq4" (OuterVolumeSpecName: "kube-api-access-nffq4") pod "569bae39-0f41-477e-955e-7b2248e16073" (UID: "569bae39-0f41-477e-955e-7b2248e16073"). InnerVolumeSpecName "kube-api-access-nffq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:56.146430 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.146387    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffq4\" (UniqueName: \"kubernetes.io/projected/569bae39-0f41-477e-955e-7b2248e16073-kube-api-access-nffq4\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:56.146430 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.146429    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569bae39-0f41-477e-955e-7b2248e16073-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:15:56.847632 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.847599    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"
Sep 09 21:15:56.848121 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.847650    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm" event={"ID":"569bae39-0f41-477e-955e-7b2248e16073","Type":"ContainerDied","Data":"5d2a7c8ba973282b24d2daf09605ed1b7126ece648f1251e5e69299362fa2726"}
Sep 09 21:15:56.848121 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.847754    2548 scope.go:117] "RemoveContainer" containerID="247a9eebb5ad0d2f117e039d2a7841671560de75124cabe5033246f28129dcb6"
Sep 09 21:15:56.870242 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.870214    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"]
Sep 09 21:15:56.873871 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:56.873848    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bd9741-764b7ddd98-hnkpm"]
Sep 09 21:15:57.970319 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:15:57.970284    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569bae39-0f41-477e-955e-7b2248e16073" path="/var/lib/kubelet/pods/569bae39-0f41-477e-955e-7b2248e16073/volumes"
Sep 09 21:16:02.110380 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:02.110333    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"]
Sep 09 21:16:02.110775 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:02.110602    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy" containerID="cri-o://4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079" gracePeriod=30
Sep 09 21:16:07.382725 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.382659    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:16:07.475233 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475189    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-data\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475248    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-token\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475281    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istiod-ca-cert\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475312    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-certs\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475375    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-socket\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475604 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475414    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrrfs\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-kube-api-access-mrrfs\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475604 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475480    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-credential-socket\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475604 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475511    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-podinfo\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475604 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475506    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:07.475604 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475544    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-envoy\") pod \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\" (UID: \"2954e2ee-da33-4e9f-a13b-c19a14822b0e\") "
Sep 09 21:16:07.475896 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475661    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:07.475896 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475716    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:07.475896 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475794    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:07.475896 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.475854    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:07.476116 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.476016    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.476116 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.476030    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.476116 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.476039    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.476116 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.476048    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.476116 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.476057    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.477529 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.477506    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:07.477678 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.477655    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:16:07.478062 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.478042    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:07.478147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.478103    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-kube-api-access-mrrfs" (OuterVolumeSpecName: "kube-api-access-mrrfs") pod "2954e2ee-da33-4e9f-a13b-c19a14822b0e" (UID: "2954e2ee-da33-4e9f-a13b-c19a14822b0e"). InnerVolumeSpecName "kube-api-access-mrrfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:07.577291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.577248    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.577291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.577282    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.577291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.577294    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.577530 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.577307    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrfs\" (UniqueName: \"kubernetes.io/projected/2954e2ee-da33-4e9f-a13b-c19a14822b0e-kube-api-access-mrrfs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:07.912872 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.912836    2548 generic.go:356] "Generic (PLEG): container finished" podID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerID="4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079" exitCode=0
Sep 09 21:16:07.913020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.912883    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"
Sep 09 21:16:07.913020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.912919    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" event={"ID":"2954e2ee-da33-4e9f-a13b-c19a14822b0e","Type":"ContainerDied","Data":"4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079"}
Sep 09 21:16:07.913020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.912995    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr" event={"ID":"2954e2ee-da33-4e9f-a13b-c19a14822b0e","Type":"ContainerDied","Data":"26faf946185199898cf5e010578bb7205a8d2db4362963af0df031924571c844"}
Sep 09 21:16:07.913020 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.913018    2548 scope.go:117] "RemoveContainer" containerID="4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079"
Sep 09 21:16:07.923064 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.923045    2548 scope.go:117] "RemoveContainer" containerID="0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"
Sep 09 21:16:07.931583 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.931565    2548 scope.go:117] "RemoveContainer" containerID="4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079"
Sep 09 21:16:07.931923 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:16:07.931900    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079\": container with ID starting with 4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079 not found: ID does not exist" containerID="4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079"
Sep 09 21:16:07.932017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.931940    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079"} err="failed to get container status \"4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079\": rpc error: code = NotFound desc = could not find container \"4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079\": container with ID starting with 4af524676d1f92d59b316dd03e356222c8d5671a2e46206d7511456c13bb2079 not found: ID does not exist"
Sep 09 21:16:07.932017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.931957    2548 scope.go:117] "RemoveContainer" containerID="0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"
Sep 09 21:16:07.932231 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:16:07.932213    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709\": container with ID starting with 0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709 not found: ID does not exist" containerID="0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"
Sep 09 21:16:07.932288 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.932236    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709"} err="failed to get container status \"0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709\": rpc error: code = NotFound desc = could not find container \"0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709\": container with ID starting with 0c89ff7ca19f555edc9006d3f5b6fd98dc6d149d4b670b1d99f31e04f322a709 not found: ID does not exist"
Sep 09 21:16:07.935665 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.935642    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"]
Sep 09 21:16:07.940047 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.940023    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bd9741-openshift-default-7fdb8d9f76-jr6qr"]
Sep 09 21:16:07.971371 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:07.971342    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" path="/var/lib/kubelet/pods/2954e2ee-da33-4e9f-a13b-c19a14822b0e/volumes"
Sep 09 21:16:13.256575 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.256542    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"]
Sep 09 21:16:13.257162 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.256834    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2" podUID="5142e403-852d-4156-ac74-0ceb369e8aff" containerName="payload-pre-processing" containerID="cri-o://319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337" gracePeriod=30
Sep 09 21:16:13.517414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.517352    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:16:13.639568 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.639537    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5142e403-852d-4156-ac74-0ceb369e8aff-plugins-config-volume\") pod \"5142e403-852d-4156-ac74-0ceb369e8aff\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") "
Sep 09 21:16:13.639806 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.639704    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m22p5\" (UniqueName: \"kubernetes.io/projected/5142e403-852d-4156-ac74-0ceb369e8aff-kube-api-access-m22p5\") pod \"5142e403-852d-4156-ac74-0ceb369e8aff\" (UID: \"5142e403-852d-4156-ac74-0ceb369e8aff\") "
Sep 09 21:16:13.640035 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.640007    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5142e403-852d-4156-ac74-0ceb369e8aff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5142e403-852d-4156-ac74-0ceb369e8aff" (UID: "5142e403-852d-4156-ac74-0ceb369e8aff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:13.641783 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.641755    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5142e403-852d-4156-ac74-0ceb369e8aff-kube-api-access-m22p5" (OuterVolumeSpecName: "kube-api-access-m22p5") pod "5142e403-852d-4156-ac74-0ceb369e8aff" (UID: "5142e403-852d-4156-ac74-0ceb369e8aff"). InnerVolumeSpecName "kube-api-access-m22p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:13.740687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.740650    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m22p5\" (UniqueName: \"kubernetes.io/projected/5142e403-852d-4156-ac74-0ceb369e8aff-kube-api-access-m22p5\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:13.740687 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.740680    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5142e403-852d-4156-ac74-0ceb369e8aff-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:13.951614 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.951574    2548 generic.go:356] "Generic (PLEG): container finished" podID="5142e403-852d-4156-ac74-0ceb369e8aff" containerID="319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337" exitCode=0
Sep 09 21:16:13.951813 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.951645    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"
Sep 09 21:16:13.951813 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.951655    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2" event={"ID":"5142e403-852d-4156-ac74-0ceb369e8aff","Type":"ContainerDied","Data":"319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337"}
Sep 09 21:16:13.951813 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.951718    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2" event={"ID":"5142e403-852d-4156-ac74-0ceb369e8aff","Type":"ContainerDied","Data":"f6120b34b346c83d82f3cf98c7007d3b48379095189334c903deb454c0f5b648"}
Sep 09 21:16:13.951813 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.951774    2548 scope.go:117] "RemoveContainer" containerID="319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337"
Sep 09 21:16:13.960940 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.960918    2548 scope.go:117] "RemoveContainer" containerID="319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337"
Sep 09 21:16:13.961240 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:16:13.961220    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337\": container with ID starting with 319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337 not found: ID does not exist" containerID="319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337"
Sep 09 21:16:13.961299 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.961249    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337"} err="failed to get container status \"319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337\": rpc error: code = NotFound desc = could not find container \"319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337\": container with ID starting with 319a0c9539e111a28e039c8b68c0b7e7996460645f71e59a79660738e4531337 not found: ID does not exist"
Sep 09 21:16:13.972360 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.972337    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"]
Sep 09 21:16:13.975993 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:13.975970    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-629537-874d6c74b-qbxz2"]
Sep 09 21:16:15.680907 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:15.680870    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"]
Sep 09 21:16:15.968368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:15.968330    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5142e403-852d-4156-ac74-0ceb369e8aff" path="/var/lib/kubelet/pods/5142e403-852d-4156-ac74-0ceb369e8aff/volumes"
Sep 09 21:16:30.702038 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:30.701989    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7" podUID="3c0b983b-4880-4b1a-80bb-293e99cc7277" containerName="main" containerID="cri-o://f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec" gracePeriod=45
Sep 09 21:16:30.962356 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:30.962288    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:16:31.010897 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.010864    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-tmp-dir\") pod \"3c0b983b-4880-4b1a-80bb-293e99cc7277\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") "
Sep 09 21:16:31.011072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.010934    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-home\") pod \"3c0b983b-4880-4b1a-80bb-293e99cc7277\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") "
Sep 09 21:16:31.011072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.010974    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0b983b-4880-4b1a-80bb-293e99cc7277-tls-certs\") pod \"3c0b983b-4880-4b1a-80bb-293e99cc7277\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") "
Sep 09 21:16:31.011072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011005    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-model-cache\") pod \"3c0b983b-4880-4b1a-80bb-293e99cc7277\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") "
Sep 09 21:16:31.011072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011044    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8bf6\" (UniqueName: \"kubernetes.io/projected/3c0b983b-4880-4b1a-80bb-293e99cc7277-kube-api-access-l8bf6\") pod \"3c0b983b-4880-4b1a-80bb-293e99cc7277\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") "
Sep 09 21:16:31.011300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011078    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-dshm\") pod \"3c0b983b-4880-4b1a-80bb-293e99cc7277\" (UID: \"3c0b983b-4880-4b1a-80bb-293e99cc7277\") "
Sep 09 21:16:31.011300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011154    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c0b983b-4880-4b1a-80bb-293e99cc7277" (UID: "3c0b983b-4880-4b1a-80bb-293e99cc7277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:31.011300 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011245    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-home" (OuterVolumeSpecName: "home") pod "3c0b983b-4880-4b1a-80bb-293e99cc7277" (UID: "3c0b983b-4880-4b1a-80bb-293e99cc7277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:31.011447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011335    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c0b983b-4880-4b1a-80bb-293e99cc7277" (UID: "3c0b983b-4880-4b1a-80bb-293e99cc7277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:31.011948 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011919    2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-tmp-dir\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.011948 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011950    2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-home\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.012162 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.011963    2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-model-cache\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.013518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.013493    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0b983b-4880-4b1a-80bb-293e99cc7277-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c0b983b-4880-4b1a-80bb-293e99cc7277" (UID: "3c0b983b-4880-4b1a-80bb-293e99cc7277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:31.013633 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.013591    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-dshm" (OuterVolumeSpecName: "dshm") pod "3c0b983b-4880-4b1a-80bb-293e99cc7277" (UID: "3c0b983b-4880-4b1a-80bb-293e99cc7277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:31.013633 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.013607    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0b983b-4880-4b1a-80bb-293e99cc7277-kube-api-access-l8bf6" (OuterVolumeSpecName: "kube-api-access-l8bf6") pod "3c0b983b-4880-4b1a-80bb-293e99cc7277" (UID: "3c0b983b-4880-4b1a-80bb-293e99cc7277"). InnerVolumeSpecName "kube-api-access-l8bf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:31.052995 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.052958    2548 generic.go:356] "Generic (PLEG): container finished" podID="3c0b983b-4880-4b1a-80bb-293e99cc7277" containerID="f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec" exitCode=0
Sep 09 21:16:31.053127 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.053003    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"
Sep 09 21:16:31.053127 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.053037    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7" event={"ID":"3c0b983b-4880-4b1a-80bb-293e99cc7277","Type":"ContainerDied","Data":"f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec"}
Sep 09 21:16:31.053127 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.053079    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7" event={"ID":"3c0b983b-4880-4b1a-80bb-293e99cc7277","Type":"ContainerDied","Data":"cf1fec7ecb422c227714eb21aa50be5a8183cf5b0225ce67a9300f7c8939a18b"}
Sep 09 21:16:31.053127 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.053096    2548 scope.go:117] "RemoveContainer" containerID="f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec"
Sep 09 21:16:31.062554 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.062537    2548 scope.go:117] "RemoveContainer" containerID="f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec"
Sep 09 21:16:31.062838 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:16:31.062819    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec\": container with ID starting with f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec not found: ID does not exist" containerID="f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec"
Sep 09 21:16:31.062902 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.062845    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec"} err="failed to get container status \"f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec\": rpc error: code = NotFound desc = could not find container \"f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec\": container with ID starting with f222abf07c2d521063333f6ea0bec1a20a707a34e8b72c280a370cf3113ba9ec not found: ID does not exist"
Sep 09 21:16:31.074654 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.074632    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"]
Sep 09 21:16:31.076714 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.076692    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3d3cac/rate-test-model-3d3cac-kserve-8594f4658c-rg4r7"]
Sep 09 21:16:31.112592 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.112573    2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0b983b-4880-4b1a-80bb-293e99cc7277-tls-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.112592 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.112594    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8bf6\" (UniqueName: \"kubernetes.io/projected/3c0b983b-4880-4b1a-80bb-293e99cc7277-kube-api-access-l8bf6\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.112724 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.112603    2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0b983b-4880-4b1a-80bb-293e99cc7277-dshm\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.220644 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.220564    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"]
Sep 09 21:16:31.220858 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.220829    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz" podUID="a3b8c94c-3016-420a-807b-d1106f3a8504" containerName="maas-api" containerID="cri-o://87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46" gracePeriod=30
Sep 09 21:16:31.282290 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.282253    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"]
Sep 09 21:16:31.282502 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.282470    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl" podUID="4d86c3f3-39dc-403e-bb23-c4659cbcce4c" containerName="payload-pre-processing" containerID="cri-o://f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428" gracePeriod=30
Sep 09 21:16:31.541816 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.541784    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:16:31.557840 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.557816    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:16:31.619782 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.619751    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-metrics-certs\") pod \"a3b8c94c-3016-420a-807b-d1106f3a8504\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") "
Sep 09 21:16:31.619966 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.619808    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-plugins-config-volume\") pod \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") "
Sep 09 21:16:31.619966 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.619930    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqpsh\" (UniqueName: \"kubernetes.io/projected/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-kube-api-access-fqpsh\") pod \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\" (UID: \"4d86c3f3-39dc-403e-bb23-c4659cbcce4c\") "
Sep 09 21:16:31.620089 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.620006    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqsw8\" (UniqueName: \"kubernetes.io/projected/a3b8c94c-3016-420a-807b-d1106f3a8504-kube-api-access-pqsw8\") pod \"a3b8c94c-3016-420a-807b-d1106f3a8504\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") "
Sep 09 21:16:31.620089 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.620037    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-maas-api-tls\") pod \"a3b8c94c-3016-420a-807b-d1106f3a8504\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") "
Sep 09 21:16:31.620198 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.620115    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b8c94c-3016-420a-807b-d1106f3a8504-openshift-service-ca\") pod \"a3b8c94c-3016-420a-807b-d1106f3a8504\" (UID: \"a3b8c94c-3016-420a-807b-d1106f3a8504\") "
Sep 09 21:16:31.620198 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.620142    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d86c3f3-39dc-403e-bb23-c4659cbcce4c" (UID: "4d86c3f3-39dc-403e-bb23-c4659cbcce4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:31.620524 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.620488    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.620656 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.620526    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b8c94c-3016-420a-807b-d1106f3a8504-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a3b8c94c-3016-420a-807b-d1106f3a8504" (UID: "a3b8c94c-3016-420a-807b-d1106f3a8504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:31.622042 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.622013    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b8c94c-3016-420a-807b-d1106f3a8504-kube-api-access-pqsw8" (OuterVolumeSpecName: "kube-api-access-pqsw8") pod "a3b8c94c-3016-420a-807b-d1106f3a8504" (UID: "a3b8c94c-3016-420a-807b-d1106f3a8504"). InnerVolumeSpecName "kube-api-access-pqsw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:31.622042 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.622033    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a3b8c94c-3016-420a-807b-d1106f3a8504" (UID: "a3b8c94c-3016-420a-807b-d1106f3a8504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:31.622164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.622044    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-kube-api-access-fqpsh" (OuterVolumeSpecName: "kube-api-access-fqpsh") pod "4d86c3f3-39dc-403e-bb23-c4659cbcce4c" (UID: "4d86c3f3-39dc-403e-bb23-c4659cbcce4c"). InnerVolumeSpecName "kube-api-access-fqpsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:31.622164 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.622063    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a3b8c94c-3016-420a-807b-d1106f3a8504" (UID: "a3b8c94c-3016-420a-807b-d1106f3a8504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:31.721175 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.721144    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqsw8\" (UniqueName: \"kubernetes.io/projected/a3b8c94c-3016-420a-807b-d1106f3a8504-kube-api-access-pqsw8\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.721175 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.721176    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-maas-api-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.721612 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.721187    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b8c94c-3016-420a-807b-d1106f3a8504-openshift-service-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.721612 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.721197    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b8c94c-3016-420a-807b-d1106f3a8504-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.721612 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.721205    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpsh\" (UniqueName: \"kubernetes.io/projected/4d86c3f3-39dc-403e-bb23-c4659cbcce4c-kube-api-access-fqpsh\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:16:31.969199 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:31.969167    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0b983b-4880-4b1a-80bb-293e99cc7277" path="/var/lib/kubelet/pods/3c0b983b-4880-4b1a-80bb-293e99cc7277/volumes"
Sep 09 21:16:32.061980 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.061944    2548 generic.go:356] "Generic (PLEG): container finished" podID="a3b8c94c-3016-420a-807b-d1106f3a8504" containerID="87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46" exitCode=0
Sep 09 21:16:32.062138 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.061988    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"
Sep 09 21:16:32.062138 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.062035    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz" event={"ID":"a3b8c94c-3016-420a-807b-d1106f3a8504","Type":"ContainerDied","Data":"87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46"}
Sep 09 21:16:32.062138 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.062099    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz" event={"ID":"a3b8c94c-3016-420a-807b-d1106f3a8504","Type":"ContainerDied","Data":"8d9f1ca44d4fa2e87c6a90a5bb805a15f0deb6bdf09b42b16f7e4501159613be"}
Sep 09 21:16:32.062138 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.062128    2548 scope.go:117] "RemoveContainer" containerID="87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46"
Sep 09 21:16:32.063540 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.063514    2548 generic.go:356] "Generic (PLEG): container finished" podID="4d86c3f3-39dc-403e-bb23-c4659cbcce4c" containerID="f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428" exitCode=0
Sep 09 21:16:32.063749 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.063602    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"
Sep 09 21:16:32.063749 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.063607    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl" event={"ID":"4d86c3f3-39dc-403e-bb23-c4659cbcce4c","Type":"ContainerDied","Data":"f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428"}
Sep 09 21:16:32.063901 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.063756    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl" event={"ID":"4d86c3f3-39dc-403e-bb23-c4659cbcce4c","Type":"ContainerDied","Data":"3b76cd1c133cffb2586d4504c6057add9d9d81dbaaee2c742805d9e5416656c2"}
Sep 09 21:16:32.071543 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.071512    2548 scope.go:117] "RemoveContainer" containerID="87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46"
Sep 09 21:16:32.071834 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:16:32.071813    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46\": container with ID starting with 87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46 not found: ID does not exist" containerID="87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46"
Sep 09 21:16:32.071908 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.071842    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46"} err="failed to get container status \"87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46\": rpc error: code = NotFound desc = could not find container \"87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46\": container with ID starting with 87fe17d64b67cd1a997e172ae0dfa2a3e96c58ff94a6ba3f409d5970cf64ac46 not found: ID does not exist"
Sep 09 21:16:32.071908 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.071858    2548 scope.go:117] "RemoveContainer" containerID="f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428"
Sep 09 21:16:32.079389 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.079370    2548 scope.go:117] "RemoveContainer" containerID="f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428"
Sep 09 21:16:32.079640 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:16:32.079621    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428\": container with ID starting with f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428 not found: ID does not exist" containerID="f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428"
Sep 09 21:16:32.079708 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.079644    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428"} err="failed to get container status \"f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428\": rpc error: code = NotFound desc = could not find container \"f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428\": container with ID starting with f6bfdbc9a54c7c4c563e856ef51892d8b111ea722a4bf709df09ee10edc59428 not found: ID does not exist"
Sep 09 21:16:32.083675 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.083654    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"]
Sep 09 21:16:32.091820 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.091800    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-459158-79c49c9fbc-84xbz"]
Sep 09 21:16:32.104555 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.104533    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"]
Sep 09 21:16:32.119071 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:32.119047    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-459158-548d8779bb-kwbjl"]
Sep 09 21:16:33.968816 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:33.968781    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d86c3f3-39dc-403e-bb23-c4659cbcce4c" path="/var/lib/kubelet/pods/4d86c3f3-39dc-403e-bb23-c4659cbcce4c/volumes"
Sep 09 21:16:33.969195 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:33.969181    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b8c94c-3016-420a-807b-d1106f3a8504" path="/var/lib/kubelet/pods/a3b8c94c-3016-420a-807b-d1106f3a8504/volumes"
Sep 09 21:16:46.584801 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.584763    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"]
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585190    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585202    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585217    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0b983b-4880-4b1a-80bb-293e99cc7277" containerName="main"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585222    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0b983b-4880-4b1a-80bb-293e99cc7277" containerName="main"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585232    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5142e403-852d-4156-ac74-0ceb369e8aff" containerName="payload-pre-processing"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585237    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="5142e403-852d-4156-ac74-0ceb369e8aff" containerName="payload-pre-processing"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585246    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy"
Sep 09 21:16:46.585251 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585251    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585257    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d86c3f3-39dc-403e-bb23-c4659cbcce4c" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585263    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d86c3f3-39dc-403e-bb23-c4659cbcce4c" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585270    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585276    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585282    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569bae39-0f41-477e-955e-7b2248e16073" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585287    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="569bae39-0f41-477e-955e-7b2248e16073" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585296    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b8c94c-3016-420a-807b-d1106f3a8504" containerName="maas-api"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585301    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b8c94c-3016-420a-807b-d1106f3a8504" containerName="maas-api"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585306    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585311    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585320    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d87220b-db55-4662-9dad-ac65fe30643f" containerName="cleanup"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585325    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d87220b-db55-4662-9dad-ac65fe30643f" containerName="cleanup"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585382    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585390    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="5142e403-852d-4156-ac74-0ceb369e8aff" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585396    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="569bae39-0f41-477e-955e-7b2248e16073" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585401    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585408    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b8c94c-3016-420a-807b-d1106f3a8504" containerName="maas-api"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585414    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d86c3f3-39dc-403e-bb23-c4659cbcce4c" containerName="payload-pre-processing"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585421    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0b983b-4880-4b1a-80bb-293e99cc7277" containerName="main"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585427    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963e4e0-e07c-4a6b-9f4c-10ca91670c65" containerName="istio-proxy"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585432    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d87220b-db55-4662-9dad-ac65fe30643f" containerName="cleanup"
Sep 09 21:16:46.585503 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.585437    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2954e2ee-da33-4e9f-a13b-c19a14822b0e" containerName="istio-proxy"
Sep 09 21:16:46.588743 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.588703    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.591017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.590996    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d7dd56-openshift-default-dockercfg-kb44w\""
Sep 09 21:16:46.599280 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.599255    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"]
Sep 09 21:16:46.668497 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668455    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-envoy\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668701 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668511    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-socket\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668701 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668603    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzrf\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-kube-api-access-xmzrf\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668851 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668706    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-credential-socket\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668851 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668809    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-certs\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668851 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668836    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-token\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668974 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668874    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istiod-ca-cert\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668974 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668898    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-podinfo\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.668974 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.668932    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-data\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770006 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.769972    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-socket\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770210 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770022    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzrf\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-kube-api-access-xmzrf\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770210 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770085    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-credential-socket\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770210 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770126    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-certs\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770210 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770142    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-token\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770210 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770163    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istiod-ca-cert\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770434    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-socket\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770483 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770452    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-certs\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770568 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770485    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-podinfo\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770568 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770538    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-data\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770645 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770575    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-credential-socket\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770645 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770610    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-envoy\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770818 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770796    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-data\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.770870 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.770826    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istiod-ca-cert\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.772717 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.772687    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-envoy\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.772860 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.772829    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-podinfo\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.777110 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.777088    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-token\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.777191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.777170    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzrf\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-kube-api-access-xmzrf\") pod \"e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") " pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:46.900011 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:46.899912    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:47.196283 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:47.196251    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"]
Sep 09 21:16:47.199149 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:16:47.199121    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173bb1c9_e6fc_481c_b4f5_d2dbd734554a.slice/crio-eb6bbc042969200047933e1b03cc135e982770cf95c9c84cec31e28da9907270 WatchSource:0}: Error finding container eb6bbc042969200047933e1b03cc135e982770cf95c9c84cec31e28da9907270: Status 404 returned error can't find the container with id eb6bbc042969200047933e1b03cc135e982770cf95c9c84cec31e28da9907270
Sep 09 21:16:47.201569 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:47.201523    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:16:47.201654 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:47.201630    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:16:47.201696 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:47.201672    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:16:48.149849 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:48.149808    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" event={"ID":"173bb1c9-e6fc-481c-b4f5-d2dbd734554a","Type":"ContainerStarted","Data":"0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"}
Sep 09 21:16:48.150225 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:48.149879    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" event={"ID":"173bb1c9-e6fc-481c-b4f5-d2dbd734554a","Type":"ContainerStarted","Data":"eb6bbc042969200047933e1b03cc135e982770cf95c9c84cec31e28da9907270"}
Sep 09 21:16:48.172316 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:48.172265    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podStartSLOduration=2.172248922 podStartE2EDuration="2.172248922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:48.169937863 +0000 UTC m=+1696.672767863" watchObservedRunningTime="2026-09-09 21:16:48.172248922 +0000 UTC m=+1696.675078923"
Sep 09 21:16:48.901002 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:48.900961    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:48.906105 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:48.906077    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:49.155248 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:49.155160    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:49.156314 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:49.156294    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:16:52.289594 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.289558    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"]
Sep 09 21:16:52.293481 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.293459    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.296656 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.296632    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d7dd56-dockercfg-zr4gc\""
Sep 09 21:16:52.296843 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.296817    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d7dd56\""
Sep 09 21:16:52.307996 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.307973    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"]
Sep 09 21:16:52.432191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.432149    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwvn\" (UniqueName: \"kubernetes.io/projected/7ea73dda-4f54-4054-96ee-fda6e452515f-kube-api-access-2wwvn\") pod \"payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.432362 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.432254    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea73dda-4f54-4054-96ee-fda6e452515f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.533436 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.533398    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea73dda-4f54-4054-96ee-fda6e452515f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.533633 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.533543    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwvn\" (UniqueName: \"kubernetes.io/projected/7ea73dda-4f54-4054-96ee-fda6e452515f-kube-api-access-2wwvn\") pod \"payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.534115 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.534090    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea73dda-4f54-4054-96ee-fda6e452515f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.543782 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.543703    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwvn\" (UniqueName: \"kubernetes.io/projected/7ea73dda-4f54-4054-96ee-fda6e452515f-kube-api-access-2wwvn\") pod \"payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.603806 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.603759    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:52.890194 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:52.889987    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"]
Sep 09 21:16:52.892892 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:16:52.892862    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea73dda_4f54_4054_96ee_fda6e452515f.slice/crio-b5c4250e081eca0179bd3e9878fba8acb4b6b20cdf0fd07d7a0f5b903f7e8f68 WatchSource:0}: Error finding container b5c4250e081eca0179bd3e9878fba8acb4b6b20cdf0fd07d7a0f5b903f7e8f68: Status 404 returned error can't find the container with id b5c4250e081eca0179bd3e9878fba8acb4b6b20cdf0fd07d7a0f5b903f7e8f68
Sep 09 21:16:53.180382 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:53.180286    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw" event={"ID":"7ea73dda-4f54-4054-96ee-fda6e452515f","Type":"ContainerStarted","Data":"9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f"}
Sep 09 21:16:53.180382 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:53.180345    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw" event={"ID":"7ea73dda-4f54-4054-96ee-fda6e452515f","Type":"ContainerStarted","Data":"b5c4250e081eca0179bd3e9878fba8acb4b6b20cdf0fd07d7a0f5b903f7e8f68"}
Sep 09 21:16:53.180382 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:53.180369    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:16:53.198399 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:16:53.198347    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw" podStartSLOduration=1.1983300909999999 podStartE2EDuration="1.198330091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:16:53.19632059 +0000 UTC m=+1701.699150591" watchObservedRunningTime="2026-09-09 21:16:53.198330091 +0000 UTC m=+1701.701160129"
Sep 09 21:17:04.187931 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:04.187889    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:17:05.196902 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:05.196858    2548 patch_prober.go:28] interesting pod/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:17:05.197383 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:05.196931    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:17:07.257377 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:07.257273    2548 generic.go:356] "Generic (PLEG): container finished" podID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerID="0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751" exitCode=0
Sep 09 21:17:07.257377 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:07.257343    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" event={"ID":"173bb1c9-e6fc-481c-b4f5-d2dbd734554a","Type":"ContainerDied","Data":"0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"}
Sep 09 21:17:07.257927 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:07.257844    2548 scope.go:117] "RemoveContainer" containerID="0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"
Sep 09 21:17:07.258982 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:07.258951    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:17:07.259101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:07.259023    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:17:07.259101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:07.259052    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:17:08.269368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:08.269321    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" event={"ID":"173bb1c9-e6fc-481c-b4f5-d2dbd734554a","Type":"ContainerStarted","Data":"e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36"}
Sep 09 21:17:08.900620 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:08.900579    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:17:08.900824 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:08.900795    2548 patch_prober.go:28] interesting pod/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 21:17:08.900875 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:08.900843    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 21:17:09.900649 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:09.900612    2548 patch_prober.go:28] interesting pod/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 21:17:09.901055 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:09.900667    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 21:17:10.901197 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:10.901164    2548 patch_prober.go:28] interesting pod/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 21:17:10.901575 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:10.901219    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 21:17:11.900901 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:11.900862    2548 patch_prober.go:28] interesting pod/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 21:17:11.901082 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:11.900915    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 21:17:12.900442 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:12.900403    2548 patch_prober.go:28] interesting pod/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 09 21:17:12.900923 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:12.900471    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 09 21:17:13.904802 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:13.904761    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:17:13.905174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:13.905040    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:17:13.905802 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:17:13.905782    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:18:28.864107 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:28.864026    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"]
Sep 09 21:18:28.867712 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:28.867694    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:28.870387 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:28.870365    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8d0f62-openshift-default-dockercfg-2v5tn\""
Sep 09 21:18:28.885424 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:28.885401    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"]
Sep 09 21:18:29.018488 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018453    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-token\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018495    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkj2\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-kube-api-access-bnkj2\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018577    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ae1def-6b39-4919-a847-13cc5ab11fff-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018612    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-data\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018657 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018636    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-podinfo\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018829 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018674    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-socket\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018829 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018711    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-credential-socket\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018829 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018760    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-envoy\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.018829 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.018798    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-certs\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.119922    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-data\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120009 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.119965    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-podinfo\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120192 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120012    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-socket\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120192 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120123    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-credential-socket\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120192 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120169    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-envoy\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120314 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120218    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-certs\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120319    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-token\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120354    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkj2\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-kube-api-access-bnkj2\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120477 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120392    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-data\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120467    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-credential-socket\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120481    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ae1def-6b39-4919-a847-13cc5ab11fff-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120538 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120357    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-socket\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.120699 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.120610    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-certs\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.121152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.121128    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ae1def-6b39-4919-a847-13cc5ab11fff-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.122426 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.122404    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-envoy\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.122653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.122635    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-podinfo\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.128049 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.128027    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkj2\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-kube-api-access-bnkj2\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.128149 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.128131    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-token\") pod \"e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") " pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.178005 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.177972    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:29.454532 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.454500    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"]
Sep 09 21:18:29.458393 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:18:29.458362    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ae1def_6b39_4919_a847_13cc5ab11fff.slice/crio-2a0b2f98236f58d067cc3d344ded06cbf08725da0e770435f8561e7ee102d38f WatchSource:0}: Error finding container 2a0b2f98236f58d067cc3d344ded06cbf08725da0e770435f8561e7ee102d38f: Status 404 returned error can't find the container with id 2a0b2f98236f58d067cc3d344ded06cbf08725da0e770435f8561e7ee102d38f
Sep 09 21:18:29.460805 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.460768    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:18:29.460945 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.460837    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:18:29.460945 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.460865    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:18:29.707836 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.707728    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" event={"ID":"15ae1def-6b39-4919-a847-13cc5ab11fff","Type":"ContainerStarted","Data":"bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"}
Sep 09 21:18:29.707836 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.707806    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" event={"ID":"15ae1def-6b39-4919-a847-13cc5ab11fff","Type":"ContainerStarted","Data":"2a0b2f98236f58d067cc3d344ded06cbf08725da0e770435f8561e7ee102d38f"}
Sep 09 21:18:29.727102 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:29.727051    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podStartSLOduration=1.727036738 podStartE2EDuration="1.727036738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:29.725821993 +0000 UTC m=+1798.228651994" watchObservedRunningTime="2026-09-09 21:18:29.727036738 +0000 UTC m=+1798.229866738"
Sep 09 21:18:30.179069 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:30.178987    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:30.184839 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:30.184809    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:30.713786 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:30.713758    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:30.714590 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:30.714566    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:32.078493 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:32.078462    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:18:32.084527 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:32.084503    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:18:33.593762 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.593714    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"]
Sep 09 21:18:33.597464 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.597442    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.600610 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.600592    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8d0f62-dockercfg-zrwzc\""
Sep 09 21:18:33.600926 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.600907    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8d0f62\""
Sep 09 21:18:33.611583 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.611558    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"]
Sep 09 21:18:33.665420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.665363    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c963e08-e975-44bf-b63d-0806b04f9297-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.665602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.665537    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqf98\" (UniqueName: \"kubernetes.io/projected/5c963e08-e975-44bf-b63d-0806b04f9297-kube-api-access-dqf98\") pod \"payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.767054 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.767012    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqf98\" (UniqueName: \"kubernetes.io/projected/5c963e08-e975-44bf-b63d-0806b04f9297-kube-api-access-dqf98\") pod \"payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.767229 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.767139    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c963e08-e975-44bf-b63d-0806b04f9297-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.767903 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.767875    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c963e08-e975-44bf-b63d-0806b04f9297-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.777814 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.777792    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqf98\" (UniqueName: \"kubernetes.io/projected/5c963e08-e975-44bf-b63d-0806b04f9297-kube-api-access-dqf98\") pod \"payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:33.907759 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:33.907669    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:34.200802 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:34.200776    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"]
Sep 09 21:18:34.203639 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:18:34.203616    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c963e08_e975_44bf_b63d_0806b04f9297.slice/crio-15d8d6ef0500015452c9cbc91aaa3c7fff37ae0b7a48a5023bae2b37b1cda367 WatchSource:0}: Error finding container 15d8d6ef0500015452c9cbc91aaa3c7fff37ae0b7a48a5023bae2b37b1cda367: Status 404 returned error can't find the container with id 15d8d6ef0500015452c9cbc91aaa3c7fff37ae0b7a48a5023bae2b37b1cda367
Sep 09 21:18:34.741174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:34.741132    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4" event={"ID":"5c963e08-e975-44bf-b63d-0806b04f9297","Type":"ContainerStarted","Data":"e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb"}
Sep 09 21:18:34.741589 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:34.741192    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4" event={"ID":"5c963e08-e975-44bf-b63d-0806b04f9297","Type":"ContainerStarted","Data":"15d8d6ef0500015452c9cbc91aaa3c7fff37ae0b7a48a5023bae2b37b1cda367"}
Sep 09 21:18:34.741589 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:34.741213    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:45.749966 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:45.749929    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:18:45.772669 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:45.772599    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4" podStartSLOduration=12.772576326 podStartE2EDuration="12.772576326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:34.768121075 +0000 UTC m=+1803.270951075" watchObservedRunningTime="2026-09-09 21:18:45.772576326 +0000 UTC m=+1814.275406328"
Sep 09 21:18:51.848845 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:51.848749    2548 generic.go:356] "Generic (PLEG): container finished" podID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerID="bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be" exitCode=0
Sep 09 21:18:51.848845 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:51.848760    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" event={"ID":"15ae1def-6b39-4919-a847-13cc5ab11fff","Type":"ContainerDied","Data":"bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"}
Sep 09 21:18:51.849316 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:51.849300    2548 scope.go:117] "RemoveContainer" containerID="bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"
Sep 09 21:18:51.850637 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:51.850608    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:18:51.850718 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:51.850684    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:18:51.850779 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:51.850718    2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:18:52.855843 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:52.855807    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" event={"ID":"15ae1def-6b39-4919-a847-13cc5ab11fff","Type":"ContainerStarted","Data":"87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3"}
Sep 09 21:18:53.178656 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:53.178552    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:53.178840 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:53.178789    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 09 21:18:53.178887 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:53.178846    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 09 21:18:54.178593 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:54.178551    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 09 21:18:54.179017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:54.178612    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 09 21:18:55.178618 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:55.178576    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 09 21:18:55.179052 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:55.178656    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 09 21:18:56.179182 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:56.179142    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 09 21:18:56.179570 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:56.179210    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 09 21:18:57.178991 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:57.178957    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 09 21:18:57.179149 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:57.179027    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 09 21:18:58.178518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:58.178478    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Sep 09 21:18:58.178932 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:58.178541    2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Sep 09 21:18:59.183007 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:59.182973    2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:59.183384 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:59.183335    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:18:59.183962 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:18:59.183943    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:19:03.428525 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.428489    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"]
Sep 09 21:19:03.429021 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.428792    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4" podUID="5c963e08-e975-44bf-b63d-0806b04f9297" containerName="payload-pre-processing" containerID="cri-o://e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb" gracePeriod=30
Sep 09 21:19:03.695747 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.695710    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:19:03.785346 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.785310    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqf98\" (UniqueName: \"kubernetes.io/projected/5c963e08-e975-44bf-b63d-0806b04f9297-kube-api-access-dqf98\") pod \"5c963e08-e975-44bf-b63d-0806b04f9297\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") "
Sep 09 21:19:03.785514 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.785357    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c963e08-e975-44bf-b63d-0806b04f9297-plugins-config-volume\") pod \"5c963e08-e975-44bf-b63d-0806b04f9297\" (UID: \"5c963e08-e975-44bf-b63d-0806b04f9297\") "
Sep 09 21:19:03.785869 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.785831    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c963e08-e975-44bf-b63d-0806b04f9297-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c963e08-e975-44bf-b63d-0806b04f9297" (UID: "5c963e08-e975-44bf-b63d-0806b04f9297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:03.787838 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.787793    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c963e08-e975-44bf-b63d-0806b04f9297-kube-api-access-dqf98" (OuterVolumeSpecName: "kube-api-access-dqf98") pod "5c963e08-e975-44bf-b63d-0806b04f9297" (UID: "5c963e08-e975-44bf-b63d-0806b04f9297"). InnerVolumeSpecName "kube-api-access-dqf98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:03.887221 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.887182    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c963e08-e975-44bf-b63d-0806b04f9297-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:03.887221 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.887214    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqf98\" (UniqueName: \"kubernetes.io/projected/5c963e08-e975-44bf-b63d-0806b04f9297-kube-api-access-dqf98\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:03.924927 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.924890    2548 generic.go:356] "Generic (PLEG): container finished" podID="5c963e08-e975-44bf-b63d-0806b04f9297" containerID="e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb" exitCode=0
Sep 09 21:19:03.924927 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.924931    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4" event={"ID":"5c963e08-e975-44bf-b63d-0806b04f9297","Type":"ContainerDied","Data":"e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb"}
Sep 09 21:19:03.925158 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.924938    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"
Sep 09 21:19:03.925158 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.924975    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4" event={"ID":"5c963e08-e975-44bf-b63d-0806b04f9297","Type":"ContainerDied","Data":"15d8d6ef0500015452c9cbc91aaa3c7fff37ae0b7a48a5023bae2b37b1cda367"}
Sep 09 21:19:03.925158 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.924999    2548 scope.go:117] "RemoveContainer" containerID="e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb"
Sep 09 21:19:03.934705 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.934683    2548 scope.go:117] "RemoveContainer" containerID="e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb"
Sep 09 21:19:03.935011 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:19:03.934990    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb\": container with ID starting with e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb not found: ID does not exist" containerID="e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb"
Sep 09 21:19:03.935075 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.935023    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb"} err="failed to get container status \"e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb\": rpc error: code = NotFound desc = could not find container \"e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb\": container with ID starting with e0cc948d80ed2f84534629b97f48513837622149d17645f7711bfcc5c10e2ceb not found: ID does not exist"
Sep 09 21:19:03.947481 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.947450    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"]
Sep 09 21:19:03.957050 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.957024    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8d0f62-65ddf664b-dwkc4"]
Sep 09 21:19:03.968441 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:03.968415    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c963e08-e975-44bf-b63d-0806b04f9297" path="/var/lib/kubelet/pods/5c963e08-e975-44bf-b63d-0806b04f9297/volumes"
Sep 09 21:19:10.456155 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:10.456118    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"]
Sep 09 21:19:10.456581 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:10.456346    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" containerID="cri-o://87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3" gracePeriod=30
Sep 09 21:19:14.185876 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:14.185836    2548 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:19:14.186318 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:14.185912    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:19:15.721457 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.721399    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:19:15.804902 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.804870    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-certs\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.804902 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.804906    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-socket\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805100 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.804992    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ae1def-6b39-4919-a847-13cc5ab11fff-istiod-ca-cert\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805100 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805023    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-token\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805133    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnkj2\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-kube-api-access-bnkj2\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805178    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-credential-socket\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805178    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.805246 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805223    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-envoy\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805467 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805284    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.805467 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805301    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-data\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805467 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805339    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-podinfo\") pod \"15ae1def-6b39-4919-a847-13cc5ab11fff\" (UID: \"15ae1def-6b39-4919-a847-13cc5ab11fff\") "
Sep 09 21:19:15.805770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805455    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.805770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805500    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ae1def-6b39-4919-a847-13cc5ab11fff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:19:15.805770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805539    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-data" (OuterVolumeSpecName: "istio-data") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.805770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805713    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.805770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805758    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.805770 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805770    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.806101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805785    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ae1def-6b39-4919-a847-13cc5ab11fff-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.806101 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.805800    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.807266 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.807241    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-token" (OuterVolumeSpecName: "istio-token") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:15.807266 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.807260    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:19:15.807414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.807255    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-kube-api-access-bnkj2" (OuterVolumeSpecName: "kube-api-access-bnkj2") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "kube-api-access-bnkj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:19:15.807414 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.807372    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15ae1def-6b39-4919-a847-13cc5ab11fff" (UID: "15ae1def-6b39-4919-a847-13cc5ab11fff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:19:15.906449 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.906415    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.906449 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.906443    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.906449 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.906452    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkj2\" (UniqueName: \"kubernetes.io/projected/15ae1def-6b39-4919-a847-13cc5ab11fff-kube-api-access-bnkj2\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.906654 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.906463    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ae1def-6b39-4919-a847-13cc5ab11fff-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:19:15.997566 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.997535    2548 generic.go:356] "Generic (PLEG): container finished" podID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerID="87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3" exitCode=0
Sep 09 21:19:15.997746 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.997581    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"
Sep 09 21:19:15.997746 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.997619    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" event={"ID":"15ae1def-6b39-4919-a847-13cc5ab11fff","Type":"ContainerDied","Data":"87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3"}
Sep 09 21:19:15.997746 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.997672    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g" event={"ID":"15ae1def-6b39-4919-a847-13cc5ab11fff","Type":"ContainerDied","Data":"2a0b2f98236f58d067cc3d344ded06cbf08725da0e770435f8561e7ee102d38f"}
Sep 09 21:19:15.997746 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:15.997695    2548 scope.go:117] "RemoveContainer" containerID="87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3"
Sep 09 21:19:16.006878 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.006859    2548 scope.go:117] "RemoveContainer" containerID="bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"
Sep 09 21:19:16.015337 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.015319    2548 scope.go:117] "RemoveContainer" containerID="87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3"
Sep 09 21:19:16.015398 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.015374    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"]
Sep 09 21:19:16.015598 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:19:16.015580    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3\": container with ID starting with 87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3 not found: ID does not exist" containerID="87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3"
Sep 09 21:19:16.015640 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.015612    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3"} err="failed to get container status \"87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3\": rpc error: code = NotFound desc = could not find container \"87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3\": container with ID starting with 87ffe01e602a777a70b9b9f01ae4823b029ca2a33454841da6e27e3b9769d9a3 not found: ID does not exist"
Sep 09 21:19:16.015640 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.015629    2548 scope.go:117] "RemoveContainer" containerID="bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"
Sep 09 21:19:16.016004 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:19:16.015986    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be\": container with ID starting with bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be not found: ID does not exist" containerID="bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"
Sep 09 21:19:16.016059 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.016010    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be"} err="failed to get container status \"bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be\": rpc error: code = NotFound desc = could not find container \"bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be\": container with ID starting with bad03a1145610ee4774df8577efbfbd2d097092b61acaad9fe516260f98039be not found: ID does not exist"
Sep 09 21:19:16.019584 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:16.019562    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8d0f62-openshift-default-59b84678bd-76v5g"]
Sep 09 21:19:17.971320 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:19:17.971279    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" path="/var/lib/kubelet/pods/15ae1def-6b39-4919-a847-13cc5ab11fff/volumes"
Sep 09 21:20:30.462250 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.462216    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"]
Sep 09 21:20:30.462795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.462449    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw" podUID="7ea73dda-4f54-4054-96ee-fda6e452515f" containerName="payload-pre-processing" containerID="cri-o://9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f" gracePeriod=30
Sep 09 21:20:30.736443 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.736416    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:20:30.874852 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.874813    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wwvn\" (UniqueName: \"kubernetes.io/projected/7ea73dda-4f54-4054-96ee-fda6e452515f-kube-api-access-2wwvn\") pod \"7ea73dda-4f54-4054-96ee-fda6e452515f\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") "
Sep 09 21:20:30.875013 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.874875    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea73dda-4f54-4054-96ee-fda6e452515f-plugins-config-volume\") pod \"7ea73dda-4f54-4054-96ee-fda6e452515f\" (UID: \"7ea73dda-4f54-4054-96ee-fda6e452515f\") "
Sep 09 21:20:30.875183 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.875160    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea73dda-4f54-4054-96ee-fda6e452515f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ea73dda-4f54-4054-96ee-fda6e452515f" (UID: "7ea73dda-4f54-4054-96ee-fda6e452515f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:30.876991 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.876951    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea73dda-4f54-4054-96ee-fda6e452515f-kube-api-access-2wwvn" (OuterVolumeSpecName: "kube-api-access-2wwvn") pod "7ea73dda-4f54-4054-96ee-fda6e452515f" (UID: "7ea73dda-4f54-4054-96ee-fda6e452515f"). InnerVolumeSpecName "kube-api-access-2wwvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:30.976848 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.976772    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea73dda-4f54-4054-96ee-fda6e452515f-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:30.976848 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:30.976806    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwvn\" (UniqueName: \"kubernetes.io/projected/7ea73dda-4f54-4054-96ee-fda6e452515f-kube-api-access-2wwvn\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:31.396518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.396430    2548 generic.go:356] "Generic (PLEG): container finished" podID="7ea73dda-4f54-4054-96ee-fda6e452515f" containerID="9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f" exitCode=0
Sep 09 21:20:31.396518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.396477    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"
Sep 09 21:20:31.396763 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.396513    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw" event={"ID":"7ea73dda-4f54-4054-96ee-fda6e452515f","Type":"ContainerDied","Data":"9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f"}
Sep 09 21:20:31.396763 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.396564    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw" event={"ID":"7ea73dda-4f54-4054-96ee-fda6e452515f","Type":"ContainerDied","Data":"b5c4250e081eca0179bd3e9878fba8acb4b6b20cdf0fd07d7a0f5b903f7e8f68"}
Sep 09 21:20:31.396763 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.396587    2548 scope.go:117] "RemoveContainer" containerID="9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f"
Sep 09 21:20:31.405501 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.405484    2548 scope.go:117] "RemoveContainer" containerID="9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f"
Sep 09 21:20:31.405790 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:20:31.405770    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f\": container with ID starting with 9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f not found: ID does not exist" containerID="9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f"
Sep 09 21:20:31.405862 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.405802    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f"} err="failed to get container status \"9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f\": rpc error: code = NotFound desc = could not find container \"9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f\": container with ID starting with 9ed406ab427edb5bc268d209a4ab5b78f53f60fff775ff4be987bdfeb3e0778f not found: ID does not exist"
Sep 09 21:20:31.417798 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.417774    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"]
Sep 09 21:20:31.419715 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.419695    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7dd56-f8b557894-4f7gw"]
Sep 09 21:20:31.969015 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:31.968986    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea73dda-4f54-4054-96ee-fda6e452515f" path="/var/lib/kubelet/pods/7ea73dda-4f54-4054-96ee-fda6e452515f/volumes"
Sep 09 21:20:37.648950 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:37.648909    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"]
Sep 09 21:20:37.649335 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:37.649119    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy" containerID="cri-o://e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36" gracePeriod=30
Sep 09 21:20:42.905661 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.905631    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:20:42.999031 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.998998    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-credential-socket\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999038    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-data\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999081    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmzrf\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-kube-api-access-xmzrf\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999101    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istiod-ca-cert\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999126    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-token\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999144    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-certs\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999162    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-socket\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999181    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-podinfo\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999212    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-envoy\") pod \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\" (UID: \"173bb1c9-e6fc-481c-b4f5-d2dbd734554a\") "
Sep 09 21:20:42.999670 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999325    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:42.999670 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999537    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:42.999670 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999554    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:42.999670 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999596    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:42.999970 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:42.999771    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:43.000038 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.000005    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-data" (OuterVolumeSpecName: "istio-data") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:43.001388 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.001362    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-kube-api-access-xmzrf" (OuterVolumeSpecName: "kube-api-access-xmzrf") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "kube-api-access-xmzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:43.001497 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.001456    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:20:43.001887 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.001860    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:43.001978 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.001904    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-token" (OuterVolumeSpecName: "istio-token") pod "173bb1c9-e6fc-481c-b4f5-d2dbd734554a" (UID: "173bb1c9-e6fc-481c-b4f5-d2dbd734554a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:43.100187 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100146    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100187 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100180    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmzrf\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-kube-api-access-xmzrf\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100187 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100191    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100200    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100207    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100215    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100222    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.100420 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.100231    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/173bb1c9-e6fc-481c-b4f5-d2dbd734554a-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:43.458899 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.458861    2548 generic.go:356] "Generic (PLEG): container finished" podID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerID="e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36" exitCode=0
Sep 09 21:20:43.459088 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.458904    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"
Sep 09 21:20:43.459088 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.458949    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" event={"ID":"173bb1c9-e6fc-481c-b4f5-d2dbd734554a","Type":"ContainerDied","Data":"e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36"}
Sep 09 21:20:43.459088 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.459007    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq" event={"ID":"173bb1c9-e6fc-481c-b4f5-d2dbd734554a","Type":"ContainerDied","Data":"eb6bbc042969200047933e1b03cc135e982770cf95c9c84cec31e28da9907270"}
Sep 09 21:20:43.459088 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.459032    2548 scope.go:117] "RemoveContainer" containerID="e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36"
Sep 09 21:20:43.470751 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.470708    2548 scope.go:117] "RemoveContainer" containerID="0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"
Sep 09 21:20:43.479218 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.479203    2548 scope.go:117] "RemoveContainer" containerID="e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36"
Sep 09 21:20:43.479470 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:20:43.479452    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36\": container with ID starting with e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36 not found: ID does not exist" containerID="e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36"
Sep 09 21:20:43.479531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.479477    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36"} err="failed to get container status \"e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36\": rpc error: code = NotFound desc = could not find container \"e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36\": container with ID starting with e09752ec1d2b867c78547cc1c40088fe05d8ddbbbc542d706b58f7f065130d36 not found: ID does not exist"
Sep 09 21:20:43.479531 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.479493    2548 scope.go:117] "RemoveContainer" containerID="0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"
Sep 09 21:20:43.479714 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:20:43.479695    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751\": container with ID starting with 0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751 not found: ID does not exist" containerID="0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"
Sep 09 21:20:43.479775 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.479719    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751"} err="failed to get container status \"0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751\": rpc error: code = NotFound desc = could not find container \"0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751\": container with ID starting with 0ddd29d1eb6c5f36ff5ab9e76e7d7aec5e164136e27dae61a5e4a40e5b9c4751 not found: ID does not exist"
Sep 09 21:20:43.483389 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.483366    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"]
Sep 09 21:20:43.488676 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.488653    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7dd56-openshift-default-6cf4d8d5dd-drfdq"]
Sep 09 21:20:43.968840 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:43.968810    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" path="/var/lib/kubelet/pods/173bb1c9-e6fc-481c-b4f5-d2dbd734554a/volumes"
Sep 09 21:20:48.593441 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.593399    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"]
Sep 09 21:20:48.593928 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.593616    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt" podUID="c7020b2d-54f1-4f85-ae7c-9dbd1895971e" containerName="maas-api" containerID="cri-o://cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368" gracePeriod=30
Sep 09 21:20:48.689408 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.689374    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"]
Sep 09 21:20:48.689619 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.689593    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg" podUID="4f875000-bff9-4b09-ace1-1f9a54f357da" containerName="payload-processing" containerID="cri-o://c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1" gracePeriod=30
Sep 09 21:20:48.859508 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.859334    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:20:48.957588 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.957558    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-maas-api-tls\") pod \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") "
Sep 09 21:20:48.957588 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.957591    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6qrk\" (UniqueName: \"kubernetes.io/projected/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-kube-api-access-d6qrk\") pod \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") "
Sep 09 21:20:48.957824 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.957658    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-openshift-service-ca\") pod \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") "
Sep 09 21:20:48.957824 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.957756    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-metrics-certs\") pod \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\" (UID: \"c7020b2d-54f1-4f85-ae7c-9dbd1895971e\") "
Sep 09 21:20:48.958046 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.958015    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7020b2d-54f1-4f85-ae7c-9dbd1895971e" (UID: "c7020b2d-54f1-4f85-ae7c-9dbd1895971e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:48.958160 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.958128    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-openshift-service-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:48.959916 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.959890    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7020b2d-54f1-4f85-ae7c-9dbd1895971e" (UID: "c7020b2d-54f1-4f85-ae7c-9dbd1895971e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:48.960017 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.959914    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7020b2d-54f1-4f85-ae7c-9dbd1895971e" (UID: "c7020b2d-54f1-4f85-ae7c-9dbd1895971e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:48.960151 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.960132    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-kube-api-access-d6qrk" (OuterVolumeSpecName: "kube-api-access-d6qrk") pod "c7020b2d-54f1-4f85-ae7c-9dbd1895971e" (UID: "c7020b2d-54f1-4f85-ae7c-9dbd1895971e"). InnerVolumeSpecName "kube-api-access-d6qrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:48.977613 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:48.977592    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:20:49.059120 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.059085    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f875000-bff9-4b09-ace1-1f9a54f357da-plugins-config-volume\") pod \"4f875000-bff9-4b09-ace1-1f9a54f357da\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") "
Sep 09 21:20:49.059311 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.059147    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll969\" (UniqueName: \"kubernetes.io/projected/4f875000-bff9-4b09-ace1-1f9a54f357da-kube-api-access-ll969\") pod \"4f875000-bff9-4b09-ace1-1f9a54f357da\" (UID: \"4f875000-bff9-4b09-ace1-1f9a54f357da\") "
Sep 09 21:20:49.059431 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.059412    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:49.059494 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.059432    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-maas-api-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:49.059494 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.059440    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qrk\" (UniqueName: \"kubernetes.io/projected/c7020b2d-54f1-4f85-ae7c-9dbd1895971e-kube-api-access-d6qrk\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:49.059590 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.059484    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f875000-bff9-4b09-ace1-1f9a54f357da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f875000-bff9-4b09-ace1-1f9a54f357da" (UID: "4f875000-bff9-4b09-ace1-1f9a54f357da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:49.061349 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.061327    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f875000-bff9-4b09-ace1-1f9a54f357da-kube-api-access-ll969" (OuterVolumeSpecName: "kube-api-access-ll969") pod "4f875000-bff9-4b09-ace1-1f9a54f357da" (UID: "4f875000-bff9-4b09-ace1-1f9a54f357da"). InnerVolumeSpecName "kube-api-access-ll969". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:49.160473 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.160383    2548 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f875000-bff9-4b09-ace1-1f9a54f357da-plugins-config-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:49.160473 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.160419    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll969\" (UniqueName: \"kubernetes.io/projected/4f875000-bff9-4b09-ace1-1f9a54f357da-kube-api-access-ll969\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:20:49.491994 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.491954    2548 generic.go:356] "Generic (PLEG): container finished" podID="4f875000-bff9-4b09-ace1-1f9a54f357da" containerID="c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1" exitCode=0
Sep 09 21:20:49.492169 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.492008    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"
Sep 09 21:20:49.492169 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.492040    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg" event={"ID":"4f875000-bff9-4b09-ace1-1f9a54f357da","Type":"ContainerDied","Data":"c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1"}
Sep 09 21:20:49.492169 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.492093    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg" event={"ID":"4f875000-bff9-4b09-ace1-1f9a54f357da","Type":"ContainerDied","Data":"b6066ef07da5028953fe03c8a1ac65a8a261697d9246e048549962399102c9d8"}
Sep 09 21:20:49.492169 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.492108    2548 scope.go:117] "RemoveContainer" containerID="c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1"
Sep 09 21:20:49.493495 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.493469    2548 generic.go:356] "Generic (PLEG): container finished" podID="c7020b2d-54f1-4f85-ae7c-9dbd1895971e" containerID="cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368" exitCode=0
Sep 09 21:20:49.493628 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.493501    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt" event={"ID":"c7020b2d-54f1-4f85-ae7c-9dbd1895971e","Type":"ContainerDied","Data":"cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368"}
Sep 09 21:20:49.493628 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.493518    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"
Sep 09 21:20:49.493628 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.493561    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt" event={"ID":"c7020b2d-54f1-4f85-ae7c-9dbd1895971e","Type":"ContainerDied","Data":"2138ba9f216c09107612c8d21f2371b2d883ffa52eb6afeb2a0192fd227dc573"}
Sep 09 21:20:49.502423 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.502407    2548 scope.go:117] "RemoveContainer" containerID="c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1"
Sep 09 21:20:49.502728 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:20:49.502697    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1\": container with ID starting with c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1 not found: ID does not exist" containerID="c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1"
Sep 09 21:20:49.502854 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.502769    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1"} err="failed to get container status \"c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1\": rpc error: code = NotFound desc = could not find container \"c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1\": container with ID starting with c2b5e5de4c838a1a5dcf431f1a9445f3633c70c2203c983c196c89b90bb5c0f1 not found: ID does not exist"
Sep 09 21:20:49.502854 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.502795    2548 scope.go:117] "RemoveContainer" containerID="cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368"
Sep 09 21:20:49.510671 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.510650    2548 scope.go:117] "RemoveContainer" containerID="cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368"
Sep 09 21:20:49.510940 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:20:49.510922    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368\": container with ID starting with cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368 not found: ID does not exist" containerID="cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368"
Sep 09 21:20:49.510990 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.510947    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368"} err="failed to get container status \"cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368\": rpc error: code = NotFound desc = could not find container \"cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368\": container with ID starting with cb0728f90edf38d230a5000ffbc2962fbd2ff0cbfbe7f913b5f5650c6ae83368 not found: ID does not exist"
Sep 09 21:20:49.534435 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.534410    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"]
Sep 09 21:20:49.541641 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.541615    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3d3cac-5b46d574d9-lgjgg"]
Sep 09 21:20:49.559258 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.559233    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"]
Sep 09 21:20:49.577113 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.577084    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3d3cac-5bc7f58556-gz5qt"]
Sep 09 21:20:49.968584 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.968551    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f875000-bff9-4b09-ace1-1f9a54f357da" path="/var/lib/kubelet/pods/4f875000-bff9-4b09-ace1-1f9a54f357da/volumes"
Sep 09 21:20:49.968974 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:49.968959    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7020b2d-54f1-4f85-ae7c-9dbd1895971e" path="/var/lib/kubelet/pods/c7020b2d-54f1-4f85-ae7c-9dbd1895971e/volumes"
Sep 09 21:20:56.748219 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:56.748171    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"]
Sep 09 21:20:56.748816 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:56.748450    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" containerID="cri-o://5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1" gracePeriod=30
Sep 09 21:20:58.859617 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:58.859576    2548 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:20:58.860037 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:20:58.859645    2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:21:01.994569 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:01.994539    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:21:02.079126 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079034    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-data\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079126 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079085    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-envoy\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079347 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079129    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b297dfb-20a2-4263-a857-52e889c86c69-istiod-ca-cert\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079347 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079162    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gkh2\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-kube-api-access-8gkh2\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079347 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079188    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-istio-token\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079347 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079307    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b297dfb-20a2-4263-a857-52e889c86c69-istio-podinfo\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079611 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079364    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:02.079611 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079397    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-credential-socket\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079611 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079471    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-certs\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079611 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079496    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-socket\") pod \"9b297dfb-20a2-4263-a857-52e889c86c69\" (UID: \"9b297dfb-20a2-4263-a857-52e889c86c69\") "
Sep 09 21:21:02.079855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079641    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:02.079855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079684    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b297dfb-20a2-4263-a857-52e889c86c69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:21:02.079855 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079698    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:02.080012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079907    2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b297dfb-20a2-4263-a857-52e889c86c69-istiod-ca-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.080012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079927    2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-credential-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.080012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079943    2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.080012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079957    2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-data\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.080012 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.079923    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:02.081500 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.081468    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b297dfb-20a2-4263-a857-52e889c86c69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:21:02.081633 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.081518    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:02.081835 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.081807    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-kube-api-access-8gkh2" (OuterVolumeSpecName: "kube-api-access-8gkh2") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "kube-api-access-8gkh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:02.081951 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.081877    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b297dfb-20a2-4263-a857-52e889c86c69" (UID: "9b297dfb-20a2-4263-a857-52e889c86c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:02.181147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.181109    2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-workload-socket\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.181147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.181138    2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b297dfb-20a2-4263-a857-52e889c86c69-istio-envoy\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.181147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.181148    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkh2\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-kube-api-access-8gkh2\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.181390 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.181157    2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b297dfb-20a2-4263-a857-52e889c86c69-istio-token\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.181390 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.181168    2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b297dfb-20a2-4263-a857-52e889c86c69-istio-podinfo\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:02.564705 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.564671    2548 generic.go:356] "Generic (PLEG): container finished" podID="9b297dfb-20a2-4263-a857-52e889c86c69" containerID="5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1" exitCode=0
Sep 09 21:21:02.564928 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.564727    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"
Sep 09 21:21:02.564928 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.564762    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" event={"ID":"9b297dfb-20a2-4263-a857-52e889c86c69","Type":"ContainerDied","Data":"5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1"}
Sep 09 21:21:02.564928 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.564816    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7" event={"ID":"9b297dfb-20a2-4263-a857-52e889c86c69","Type":"ContainerDied","Data":"c31cee9586df073edd5de02cba93dd7f85aaa0027833683c87795f2a7b8def9c"}
Sep 09 21:21:02.564928 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.564839    2548 scope.go:117] "RemoveContainer" containerID="5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1"
Sep 09 21:21:02.573990 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.573970    2548 scope.go:117] "RemoveContainer" containerID="fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"
Sep 09 21:21:02.581834 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.581815    2548 scope.go:117] "RemoveContainer" containerID="5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1"
Sep 09 21:21:02.582150 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:21:02.582122    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1\": container with ID starting with 5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1 not found: ID does not exist" containerID="5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1"
Sep 09 21:21:02.582254 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.582154    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1"} err="failed to get container status \"5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1\": rpc error: code = NotFound desc = could not find container \"5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1\": container with ID starting with 5214ba05bbf454d49aab6e5d4cbe3ae79afb51cf02bae62b7354546d7abc1ef1 not found: ID does not exist"
Sep 09 21:21:02.582254 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.582173    2548 scope.go:117] "RemoveContainer" containerID="fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"
Sep 09 21:21:02.582443 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:21:02.582425    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255\": container with ID starting with fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255 not found: ID does not exist" containerID="fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"
Sep 09 21:21:02.582489 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.582448    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255"} err="failed to get container status \"fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255\": rpc error: code = NotFound desc = could not find container \"fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255\": container with ID starting with fb3b7f049fd67d29dca3d04f3859aea646d4f40b0997847749dc04d0c3114255 not found: ID does not exist"
Sep 09 21:21:02.589835 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.589808    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"]
Sep 09 21:21:02.594351 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:02.594319    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3d3cac-openshift-default-7655f6d874-6g8s7"]
Sep 09 21:21:03.968050 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:03.968011    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" path="/var/lib/kubelet/pods/9b297dfb-20a2-4263-a857-52e889c86c69/volumes"
Sep 09 21:21:07.731011 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:07.730977    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"]
Sep 09 21:21:07.735000 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:07.734969    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8db5a7-29816475-q9kjr"]
Sep 09 21:21:07.968419 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:07.968386    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d87220b-db55-4662-9dad-ac65fe30643f" path="/var/lib/kubelet/pods/6d87220b-db55-4662-9dad-ac65fe30643f/volumes"
Sep 09 21:21:17.998360 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:17.998329    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44688b45-2nrvb"]
Sep 09 21:21:17.998864 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:17.998528    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f44688b45-2nrvb" podUID="0119f136-b1b4-4306-879f-825cdf185435" containerName="manager" containerID="cri-o://0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d" gracePeriod=30
Sep 09 21:21:18.250807 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.250727    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:21:18.346191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.346156    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdxpz\" (UniqueName: \"kubernetes.io/projected/0119f136-b1b4-4306-879f-825cdf185435-kube-api-access-sdxpz\") pod \"0119f136-b1b4-4306-879f-825cdf185435\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") "
Sep 09 21:21:18.346355 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.346260    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-metrics-certs\") pod \"0119f136-b1b4-4306-879f-825cdf185435\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") "
Sep 09 21:21:18.346355 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.346301    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-webhook-cert\") pod \"0119f136-b1b4-4306-879f-825cdf185435\" (UID: \"0119f136-b1b4-4306-879f-825cdf185435\") "
Sep 09 21:21:18.348579 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.348533    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0119f136-b1b4-4306-879f-825cdf185435" (UID: "0119f136-b1b4-4306-879f-825cdf185435"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:21:18.348892 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.348865    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0119f136-b1b4-4306-879f-825cdf185435" (UID: "0119f136-b1b4-4306-879f-825cdf185435"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:21:18.349144 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.349123    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0119f136-b1b4-4306-879f-825cdf185435-kube-api-access-sdxpz" (OuterVolumeSpecName: "kube-api-access-sdxpz") pod "0119f136-b1b4-4306-879f-825cdf185435" (UID: "0119f136-b1b4-4306-879f-825cdf185435"). InnerVolumeSpecName "kube-api-access-sdxpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:18.447821 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.447782    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:18.447821 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.447815    2548 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0119f136-b1b4-4306-879f-825cdf185435-webhook-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:18.448022 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.447836    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxpz\" (UniqueName: \"kubernetes.io/projected/0119f136-b1b4-4306-879f-825cdf185435-kube-api-access-sdxpz\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:21:18.648603 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.648508    2548 generic.go:356] "Generic (PLEG): container finished" podID="0119f136-b1b4-4306-879f-825cdf185435" containerID="0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d" exitCode=0
Sep 09 21:21:18.648603 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.648556    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-2nrvb"
Sep 09 21:21:18.648603 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.648585    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-2nrvb" event={"ID":"0119f136-b1b4-4306-879f-825cdf185435","Type":"ContainerDied","Data":"0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d"}
Sep 09 21:21:18.648926 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.648639    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-2nrvb" event={"ID":"0119f136-b1b4-4306-879f-825cdf185435","Type":"ContainerDied","Data":"4890345f42187b990ef3ced1c9387c0dbf5a2f365a2af3aa9dcd29d5826fed49"}
Sep 09 21:21:18.648926 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.648669    2548 scope.go:117] "RemoveContainer" containerID="0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d"
Sep 09 21:21:18.658551 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.658530    2548 scope.go:117] "RemoveContainer" containerID="0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d"
Sep 09 21:21:18.658827 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:21:18.658803    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d\": container with ID starting with 0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d not found: ID does not exist" containerID="0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d"
Sep 09 21:21:18.658896 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.658834    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d"} err="failed to get container status \"0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d\": rpc error: code = NotFound desc = could not find container \"0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d\": container with ID starting with 0fef0f2ef25c2178aa353db1f7aa9ba4a95c39411c2e6b0dc3b405593a34320d not found: ID does not exist"
Sep 09 21:21:18.674650 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.674616    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44688b45-2nrvb"]
Sep 09 21:21:18.679180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:18.679159    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f44688b45-2nrvb"]
Sep 09 21:21:19.969165 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:19.969132    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0119f136-b1b4-4306-879f-825cdf185435" path="/var/lib/kubelet/pods/0119f136-b1b4-4306-879f-825cdf185435/volumes"
Sep 09 21:21:32.266949 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:21:32.266872    2548 scope.go:117] "RemoveContainer" containerID="c24fd1a6090de89c6acb079fe8eee4accf68a0e12d180b9502d71d54597b7303"
Sep 09 21:22:19.496180 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496147    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f44688b45-hjxxh"]
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496546    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496560    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496570    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7020b2d-54f1-4f85-ae7c-9dbd1895971e" containerName="maas-api"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496576    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7020b2d-54f1-4f85-ae7c-9dbd1895971e" containerName="maas-api"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496582    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496588    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496596    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c963e08-e975-44bf-b63d-0806b04f9297" containerName="payload-pre-processing"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496602    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c963e08-e975-44bf-b63d-0806b04f9297" containerName="payload-pre-processing"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496615    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496620    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496631    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea73dda-4f54-4054-96ee-fda6e452515f" containerName="payload-pre-processing"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496636    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea73dda-4f54-4054-96ee-fda6e452515f" containerName="payload-pre-processing"
Sep 09 21:22:19.496638 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496642    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496648    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496655    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0119f136-b1b4-4306-879f-825cdf185435" containerName="manager"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496660    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0119f136-b1b4-4306-879f-825cdf185435" containerName="manager"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496666    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f875000-bff9-4b09-ace1-1f9a54f357da" containerName="payload-processing"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496673    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f875000-bff9-4b09-ace1-1f9a54f357da" containerName="payload-processing"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496743    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496753    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496762    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f875000-bff9-4b09-ace1-1f9a54f357da" containerName="payload-processing"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496775    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c963e08-e975-44bf-b63d-0806b04f9297" containerName="payload-pre-processing"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496785    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496795    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496801    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0119f136-b1b4-4306-879f-825cdf185435" containerName="manager"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496806    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea73dda-4f54-4054-96ee-fda6e452515f" containerName="payload-pre-processing"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496811    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy"
Sep 09 21:22:19.497072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.496817    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7020b2d-54f1-4f85-ae7c-9dbd1895971e" containerName="maas-api"
Sep 09 21:22:19.499801 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.499782    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.503746 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.503705    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-gateway-identity\""
Sep 09 21:22:19.504583 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.504559    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:22:19.504690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.504610    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:22:19.504690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.504636    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:22:19.504690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.504661    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f842q\""
Sep 09 21:22:19.513794 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.513769    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44688b45-hjxxh"]
Sep 09 21:22:19.613821 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.613783    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-webhook-cert\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.613821 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.613817    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-metrics-certs\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.614032 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.613835    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vf6\" (UniqueName: \"kubernetes.io/projected/40f5ee83-e72c-4eb2-9df9-26474395a9f3-kube-api-access-d6vf6\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.714590 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.714556    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-webhook-cert\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.714590 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.714593    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-metrics-certs\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.714807 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.714611    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vf6\" (UniqueName: \"kubernetes.io/projected/40f5ee83-e72c-4eb2-9df9-26474395a9f3-kube-api-access-d6vf6\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.717322 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.717296    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-webhook-cert\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.717447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.717304    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-metrics-certs\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.723911 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.723890    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vf6\" (UniqueName: \"kubernetes.io/projected/40f5ee83-e72c-4eb2-9df9-26474395a9f3-kube-api-access-d6vf6\") pod \"maas-controller-f44688b45-hjxxh\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") " pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:19.810702 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:19.810625    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:20.089054 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:20.088978    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44688b45-hjxxh"]
Sep 09 21:22:20.091425 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:22:20.091394    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f5ee83_e72c_4eb2_9df9_26474395a9f3.slice/crio-a8d0879e3c668733cf5664b53255729cb1639d84b96a7b23f6c52fd6c0919375 WatchSource:0}: Error finding container a8d0879e3c668733cf5664b53255729cb1639d84b96a7b23f6c52fd6c0919375: Status 404 returned error can't find the container with id a8d0879e3c668733cf5664b53255729cb1639d84b96a7b23f6c52fd6c0919375
Sep 09 21:22:20.092772 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:20.092753    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:22:20.961408 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:20.961375    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-hjxxh" event={"ID":"40f5ee83-e72c-4eb2-9df9-26474395a9f3","Type":"ContainerStarted","Data":"9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12"}
Sep 09 21:22:20.961831 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:20.961433    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-hjxxh" event={"ID":"40f5ee83-e72c-4eb2-9df9-26474395a9f3","Type":"ContainerStarted","Data":"a8d0879e3c668733cf5664b53255729cb1639d84b96a7b23f6c52fd6c0919375"}
Sep 09 21:22:20.961831 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:20.961473    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:20.982218 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:20.982171    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f44688b45-hjxxh" podStartSLOduration=1.467357925 podStartE2EDuration="1.982154412s" totalImagesPullingTime="514.796487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:20.98035615 +0000 UTC m=+2029.483186150" watchObservedRunningTime="2026-09-09 21:22:20.982154412 +0000 UTC m=+2029.484984412"
Sep 09 21:22:31.969176 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:31.969146    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:43.430833 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.430796    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44688b45-hjxxh"]
Sep 09 21:22:43.431294 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.431051    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f44688b45-hjxxh" podUID="40f5ee83-e72c-4eb2-9df9-26474395a9f3" containerName="manager" containerID="cri-o://9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12" gracePeriod=30
Sep 09 21:22:43.689853 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.689815    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:43.837191 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.837155    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-metrics-certs\") pod \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") "
Sep 09 21:22:43.837370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.837206    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6vf6\" (UniqueName: \"kubernetes.io/projected/40f5ee83-e72c-4eb2-9df9-26474395a9f3-kube-api-access-d6vf6\") pod \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") "
Sep 09 21:22:43.837370 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.837231    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-webhook-cert\") pod \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\" (UID: \"40f5ee83-e72c-4eb2-9df9-26474395a9f3\") "
Sep 09 21:22:43.839318 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.839281    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f5ee83-e72c-4eb2-9df9-26474395a9f3-kube-api-access-d6vf6" (OuterVolumeSpecName: "kube-api-access-d6vf6") pod "40f5ee83-e72c-4eb2-9df9-26474395a9f3" (UID: "40f5ee83-e72c-4eb2-9df9-26474395a9f3"). InnerVolumeSpecName "kube-api-access-d6vf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:43.839439 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.839310    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40f5ee83-e72c-4eb2-9df9-26474395a9f3" (UID: "40f5ee83-e72c-4eb2-9df9-26474395a9f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:43.839439 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.839353    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "40f5ee83-e72c-4eb2-9df9-26474395a9f3" (UID: "40f5ee83-e72c-4eb2-9df9-26474395a9f3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:43.938623 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.938583    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:22:43.938623 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.938618    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vf6\" (UniqueName: \"kubernetes.io/projected/40f5ee83-e72c-4eb2-9df9-26474395a9f3-kube-api-access-d6vf6\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:22:43.938847 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:43.938632    2548 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40f5ee83-e72c-4eb2-9df9-26474395a9f3-webhook-cert\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:22:44.076447 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.076405    2548 generic.go:356] "Generic (PLEG): container finished" podID="40f5ee83-e72c-4eb2-9df9-26474395a9f3" containerID="9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12" exitCode=0
Sep 09 21:22:44.076613 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.076493    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-hjxxh"
Sep 09 21:22:44.076613 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.076532    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-hjxxh" event={"ID":"40f5ee83-e72c-4eb2-9df9-26474395a9f3","Type":"ContainerDied","Data":"9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12"}
Sep 09 21:22:44.076613 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.076588    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-hjxxh" event={"ID":"40f5ee83-e72c-4eb2-9df9-26474395a9f3","Type":"ContainerDied","Data":"a8d0879e3c668733cf5664b53255729cb1639d84b96a7b23f6c52fd6c0919375"}
Sep 09 21:22:44.076727 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.076618    2548 scope.go:117] "RemoveContainer" containerID="9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12"
Sep 09 21:22:44.085663 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.085641    2548 scope.go:117] "RemoveContainer" containerID="9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12"
Sep 09 21:22:44.085971 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:22:44.085952    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12\": container with ID starting with 9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12 not found: ID does not exist" containerID="9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12"
Sep 09 21:22:44.086022 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.085979    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12"} err="failed to get container status \"9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12\": rpc error: code = NotFound desc = could not find container \"9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12\": container with ID starting with 9aa5c4c325bd8f01f40ee3f298aac759b85663c14aed5d2f17e00b1507997e12 not found: ID does not exist"
Sep 09 21:22:44.105806 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.105775    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f44688b45-hjxxh"]
Sep 09 21:22:44.111352 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.111326    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f44688b45-hjxxh"]
Sep 09 21:22:44.924700 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.924663    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f44688b45-47c9d"]
Sep 09 21:22:44.925106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925056    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy"
Sep 09 21:22:44.925106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925068    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b297dfb-20a2-4263-a857-52e889c86c69" containerName="istio-proxy"
Sep 09 21:22:44.925106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925075    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy"
Sep 09 21:22:44.925106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925080    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="173bb1c9-e6fc-481c-b4f5-d2dbd734554a" containerName="istio-proxy"
Sep 09 21:22:44.925106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925098    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40f5ee83-e72c-4eb2-9df9-26474395a9f3" containerName="manager"
Sep 09 21:22:44.925106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925103    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="40f5ee83-e72c-4eb2-9df9-26474395a9f3" containerName="manager"
Sep 09 21:22:44.925293 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925167    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="40f5ee83-e72c-4eb2-9df9-26474395a9f3" containerName="manager"
Sep 09 21:22:44.925293 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.925174    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ae1def-6b39-4919-a847-13cc5ab11fff" containerName="istio-proxy"
Sep 09 21:22:44.929444 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.929423    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:44.933906 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.933883    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:22:44.934056 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.933922    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f842q\""
Sep 09 21:22:44.934056 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.933938    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-gateway-identity\""
Sep 09 21:22:44.934056 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.933927    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:22:44.934056 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.934008    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:22:44.945168 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:44.945141    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44688b45-47c9d"]
Sep 09 21:22:45.049117 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.049079    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05314fe6-a0c2-401d-b453-0126895805c7-metrics-certs\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.049281 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.049210    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmxb\" (UniqueName: \"kubernetes.io/projected/05314fe6-a0c2-401d-b453-0126895805c7-kube-api-access-vkmxb\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.049517 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.049492    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05314fe6-a0c2-401d-b453-0126895805c7-webhook-cert\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.150092 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.150054    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05314fe6-a0c2-401d-b453-0126895805c7-webhook-cert\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.150274 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.150118    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05314fe6-a0c2-401d-b453-0126895805c7-metrics-certs\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.150274 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.150138    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmxb\" (UniqueName: \"kubernetes.io/projected/05314fe6-a0c2-401d-b453-0126895805c7-kube-api-access-vkmxb\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.152533 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.152500    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05314fe6-a0c2-401d-b453-0126895805c7-webhook-cert\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.152678 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.152654    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05314fe6-a0c2-401d-b453-0126895805c7-metrics-certs\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.161233 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.161205    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmxb\" (UniqueName: \"kubernetes.io/projected/05314fe6-a0c2-401d-b453-0126895805c7-kube-api-access-vkmxb\") pod \"maas-controller-f44688b45-47c9d\" (UID: \"05314fe6-a0c2-401d-b453-0126895805c7\") " pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.240153 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.240114    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:45.518985 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.518955    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f44688b45-47c9d"]
Sep 09 21:22:45.521214 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:22:45.521187    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05314fe6_a0c2_401d_b453_0126895805c7.slice/crio-e295fd3a7d927ddd7536af8cd30281a10c56ad949b19bc4246c98e319afb6c97 WatchSource:0}: Error finding container e295fd3a7d927ddd7536af8cd30281a10c56ad949b19bc4246c98e319afb6c97: Status 404 returned error can't find the container with id e295fd3a7d927ddd7536af8cd30281a10c56ad949b19bc4246c98e319afb6c97
Sep 09 21:22:45.970025 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:45.969998    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40f5ee83-e72c-4eb2-9df9-26474395a9f3" path="/var/lib/kubelet/pods/40f5ee83-e72c-4eb2-9df9-26474395a9f3/volumes"
Sep 09 21:22:46.089201 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:46.089167    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-47c9d" event={"ID":"05314fe6-a0c2-401d-b453-0126895805c7","Type":"ContainerStarted","Data":"33e05f185593a8cd6e128f94ebf92aa41faad5eb091b6bd5706367b0e2d0436e"}
Sep 09 21:22:46.089368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:46.089219    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f44688b45-47c9d" event={"ID":"05314fe6-a0c2-401d-b453-0126895805c7","Type":"ContainerStarted","Data":"e295fd3a7d927ddd7536af8cd30281a10c56ad949b19bc4246c98e319afb6c97"}
Sep 09 21:22:46.089368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:46.089305    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:22:46.111412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:46.111360    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f44688b45-47c9d" podStartSLOduration=1.719147647 podStartE2EDuration="2.111344925s" totalImagesPullingTime="392.197278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:46.108771375 +0000 UTC m=+2054.611601367" watchObservedRunningTime="2026-09-09 21:22:46.111344925 +0000 UTC m=+2054.614174927"
Sep 09 21:22:57.096889 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:22:57.096808    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f44688b45-47c9d"
Sep 09 21:23:32.111371 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:32.111342    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:23:32.117137 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:32.117114    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:23:52.213521 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:52.213484    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"]
Sep 09 21:23:52.216006 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:52.213708    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk" podUID="5dc2b8c5-5c34-4c0e-846d-0d3e82688217" containerName="manager" containerID="cri-o://a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2" gracePeriod=10
Sep 09 21:23:53.067518 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.067488    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 21:23:53.173373 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.173342    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-extensions-socket-volume\") pod \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") "
Sep 09 21:23:53.173544 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.173476    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk4wr\" (UniqueName: \"kubernetes.io/projected/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-kube-api-access-xk4wr\") pod \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\" (UID: \"5dc2b8c5-5c34-4c0e-846d-0d3e82688217\") "
Sep 09 21:23:53.173863 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.173831    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5dc2b8c5-5c34-4c0e-846d-0d3e82688217" (UID: "5dc2b8c5-5c34-4c0e-846d-0d3e82688217"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:53.175479 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.175455    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-kube-api-access-xk4wr" (OuterVolumeSpecName: "kube-api-access-xk4wr") pod "5dc2b8c5-5c34-4c0e-846d-0d3e82688217" (UID: "5dc2b8c5-5c34-4c0e-846d-0d3e82688217"). InnerVolumeSpecName "kube-api-access-xk4wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:53.274363 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.274328    2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-extensions-socket-volume\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:23:53.274363 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.274357    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4wr\" (UniqueName: \"kubernetes.io/projected/5dc2b8c5-5c34-4c0e-846d-0d3e82688217-kube-api-access-xk4wr\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:23:53.424982 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.424944    2548 generic.go:356] "Generic (PLEG): container finished" podID="5dc2b8c5-5c34-4c0e-846d-0d3e82688217" containerID="a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2" exitCode=0
Sep 09 21:23:53.425178 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.424993    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"
Sep 09 21:23:53.425178 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.425026    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk" event={"ID":"5dc2b8c5-5c34-4c0e-846d-0d3e82688217","Type":"ContainerDied","Data":"a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2"}
Sep 09 21:23:53.425178 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.425078    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk" event={"ID":"5dc2b8c5-5c34-4c0e-846d-0d3e82688217","Type":"ContainerDied","Data":"826622e4238b84e5f4a6727af20978ddaea9b2b47ebf6820b6d7cd2495c8336a"}
Sep 09 21:23:53.425178 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.425101    2548 scope.go:117] "RemoveContainer" containerID="a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2"
Sep 09 21:23:53.436647 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.436622    2548 scope.go:117] "RemoveContainer" containerID="a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2"
Sep 09 21:23:53.436953 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:23:53.436934    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2\": container with ID starting with a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2 not found: ID does not exist" containerID="a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2"
Sep 09 21:23:53.437011 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.436962    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2"} err="failed to get container status \"a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2\": rpc error: code = NotFound desc = could not find container \"a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2\": container with ID starting with a84c212a0b1e9f23519eb8e1929f2b8249ef142e5755d196d975723257eb63d2 not found: ID does not exist"
Sep 09 21:23:53.451787 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.451765    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"]
Sep 09 21:23:53.455598 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.455570    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-j56qk"]
Sep 09 21:23:53.968461 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:23:53.968426    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc2b8c5-5c34-4c0e-846d-0d3e82688217" path="/var/lib/kubelet/pods/5dc2b8c5-5c34-4c0e-846d-0d3e82688217/volumes"
Sep 09 21:24:52.932644 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.932608    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"]
Sep 09 21:24:52.933142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.933012    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2b8c5-5c34-4c0e-846d-0d3e82688217" containerName="manager"
Sep 09 21:24:52.933142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.933023    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2b8c5-5c34-4c0e-846d-0d3e82688217" containerName="manager"
Sep 09 21:24:52.933142 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.933093    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2b8c5-5c34-4c0e-846d-0d3e82688217" containerName="manager"
Sep 09 21:24:52.936174 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.936156    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:52.939278 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.939259    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ks4mt\""
Sep 09 21:24:52.960715 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:52.960690    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"]
Sep 09 21:24:53.005660 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.005628    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9qrw\" (UniqueName: \"kubernetes.io/projected/f41ba5bb-2e87-474d-a7f6-a1057a2d881a-kube-api-access-g9qrw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vdzvc\" (UID: \"f41ba5bb-2e87-474d-a7f6-a1057a2d881a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.005817 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.005669    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f41ba5bb-2e87-474d-a7f6-a1057a2d881a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vdzvc\" (UID: \"f41ba5bb-2e87-474d-a7f6-a1057a2d881a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.106611 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.106573    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9qrw\" (UniqueName: \"kubernetes.io/projected/f41ba5bb-2e87-474d-a7f6-a1057a2d881a-kube-api-access-g9qrw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vdzvc\" (UID: \"f41ba5bb-2e87-474d-a7f6-a1057a2d881a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.106611 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.106613    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f41ba5bb-2e87-474d-a7f6-a1057a2d881a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vdzvc\" (UID: \"f41ba5bb-2e87-474d-a7f6-a1057a2d881a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.107016 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.106995    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f41ba5bb-2e87-474d-a7f6-a1057a2d881a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vdzvc\" (UID: \"f41ba5bb-2e87-474d-a7f6-a1057a2d881a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.119484 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.119453    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9qrw\" (UniqueName: \"kubernetes.io/projected/f41ba5bb-2e87-474d-a7f6-a1057a2d881a-kube-api-access-g9qrw\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vdzvc\" (UID: \"f41ba5bb-2e87-474d-a7f6-a1057a2d881a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.245669 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.245634    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.539089 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:24:53.539042    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41ba5bb_2e87_474d_a7f6_a1057a2d881a.slice/crio-a5fff16e7bf6c03e07a11fa1b85a80f4b88bfe04d72c1bffebbac564d23f9093 WatchSource:0}: Error finding container a5fff16e7bf6c03e07a11fa1b85a80f4b88bfe04d72c1bffebbac564d23f9093: Status 404 returned error can't find the container with id a5fff16e7bf6c03e07a11fa1b85a80f4b88bfe04d72c1bffebbac564d23f9093
Sep 09 21:24:53.540880 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.540853    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"]
Sep 09 21:24:53.712104 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.712056    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc" event={"ID":"f41ba5bb-2e87-474d-a7f6-a1057a2d881a","Type":"ContainerStarted","Data":"8340d149ad25e949535b2b73e6644c304e2274aaff72f17f39fe25dbb4d1f0fa"}
Sep 09 21:24:53.712291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.712115    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc" event={"ID":"f41ba5bb-2e87-474d-a7f6-a1057a2d881a","Type":"ContainerStarted","Data":"a5fff16e7bf6c03e07a11fa1b85a80f4b88bfe04d72c1bffebbac564d23f9093"}
Sep 09 21:24:53.712291 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.712219    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:24:53.743963 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:24:53.743898    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc" podStartSLOduration=1.743881193 podStartE2EDuration="1.743881193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:24:53.742500716 +0000 UTC m=+2182.245330755" watchObservedRunningTime="2026-09-09 21:24:53.743881193 +0000 UTC m=+2182.246711194"
Sep 09 21:25:04.719241 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:25:04.719206    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vdzvc"
Sep 09 21:26:30.148936 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.148900    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-584858f848-7kkvc"]
Sep 09 21:26:30.152828 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.152787    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.164803 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.164710    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-584858f848-7kkvc"]
Sep 09 21:26:30.206402 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.206366    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0653f804-00a7-4e88-85fb-be8f56ce4be8-maas-api-tls\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.206555 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.206447    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0653f804-00a7-4e88-85fb-be8f56ce4be8-openshift-service-ca\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.206555 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.206516    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0653f804-00a7-4e88-85fb-be8f56ce4be8-metrics-certs\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.206555 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.206544    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnxw\" (UniqueName: \"kubernetes.io/projected/0653f804-00a7-4e88-85fb-be8f56ce4be8-kube-api-access-wxnxw\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.307966 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.307927    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0653f804-00a7-4e88-85fb-be8f56ce4be8-maas-api-tls\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.308152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.307994    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0653f804-00a7-4e88-85fb-be8f56ce4be8-openshift-service-ca\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.308152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.308054    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0653f804-00a7-4e88-85fb-be8f56ce4be8-metrics-certs\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.308152 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.308080    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnxw\" (UniqueName: \"kubernetes.io/projected/0653f804-00a7-4e88-85fb-be8f56ce4be8-kube-api-access-wxnxw\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.308694 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.308671    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0653f804-00a7-4e88-85fb-be8f56ce4be8-openshift-service-ca\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.310326 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.310302    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0653f804-00a7-4e88-85fb-be8f56ce4be8-maas-api-tls\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.310428 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.310409    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0653f804-00a7-4e88-85fb-be8f56ce4be8-metrics-certs\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.315763 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.315721    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnxw\" (UniqueName: \"kubernetes.io/projected/0653f804-00a7-4e88-85fb-be8f56ce4be8-kube-api-access-wxnxw\") pod \"maas-api-584858f848-7kkvc\" (UID: \"0653f804-00a7-4e88-85fb-be8f56ce4be8\") " pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.463539 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.463502    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:30.744674 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:30.744642    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-584858f848-7kkvc"]
Sep 09 21:26:31.118110 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:31.118087    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-gateway-identity\""
Sep 09 21:26:31.206090 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:31.206049    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc" event={"ID":"0653f804-00a7-4e88-85fb-be8f56ce4be8","Type":"ContainerStarted","Data":"755eade52b02f3c282d597743187ff5b5aa8b5904153dad2c33378a2dc8fae16"}
Sep 09 21:26:31.206498 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:31.206113    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc" event={"ID":"0653f804-00a7-4e88-85fb-be8f56ce4be8","Type":"ContainerStarted","Data":"d38fb15b553e8c73aeb60bb50ae9cce024520e74f7825d4d2a5f9ecb939f10ac"}
Sep 09 21:26:31.206498 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:31.206136    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:31.233356 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:31.233292    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc" podStartSLOduration=0.865875198 podStartE2EDuration="1.233273629s" totalImagesPullingTime="367.398431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:26:31.231184317 +0000 UTC m=+2279.734014320" watchObservedRunningTime="2026-09-09 21:26:31.233273629 +0000 UTC m=+2279.736103632"
Sep 09 21:26:37.216768 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:37.216715    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-584858f848-7kkvc"
Sep 09 21:26:45.480925 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:45.480888    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-p8md7"]
Sep 09 21:26:45.481585 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:45.481136    2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7" podUID="65f79f8e-ebc2-43a5-b8b5-72400aa31111" containerName="maas-api" containerID="cri-o://18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15" gracePeriod=30
Sep 09 21:26:46.729501 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.729476    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:26:46.872914 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.872831    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-metrics-certs\") pod \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") "
Sep 09 21:26:46.872914 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.872872    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p9m4\" (UniqueName: \"kubernetes.io/projected/65f79f8e-ebc2-43a5-b8b5-72400aa31111-kube-api-access-5p9m4\") pod \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") "
Sep 09 21:26:46.872914 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.872903    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls\") pod \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") "
Sep 09 21:26:46.873199 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.872962    2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f79f8e-ebc2-43a5-b8b5-72400aa31111-openshift-service-ca\") pod \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\" (UID: \"65f79f8e-ebc2-43a5-b8b5-72400aa31111\") "
Sep 09 21:26:46.873368 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.873335    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f79f8e-ebc2-43a5-b8b5-72400aa31111-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65f79f8e-ebc2-43a5-b8b5-72400aa31111" (UID: "65f79f8e-ebc2-43a5-b8b5-72400aa31111"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:26:46.875080 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.875045    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f79f8e-ebc2-43a5-b8b5-72400aa31111-kube-api-access-5p9m4" (OuterVolumeSpecName: "kube-api-access-5p9m4") pod "65f79f8e-ebc2-43a5-b8b5-72400aa31111" (UID: "65f79f8e-ebc2-43a5-b8b5-72400aa31111"). InnerVolumeSpecName "kube-api-access-5p9m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:26:46.875080 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.875045    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65f79f8e-ebc2-43a5-b8b5-72400aa31111" (UID: "65f79f8e-ebc2-43a5-b8b5-72400aa31111"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:26:46.875223 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.875087    2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65f79f8e-ebc2-43a5-b8b5-72400aa31111" (UID: "65f79f8e-ebc2-43a5-b8b5-72400aa31111"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:26:46.974154 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.974118    2548 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-metrics-certs\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:26:46.974154 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.974150    2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p9m4\" (UniqueName: \"kubernetes.io/projected/65f79f8e-ebc2-43a5-b8b5-72400aa31111-kube-api-access-5p9m4\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:26:46.974154 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.974161    2548 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f79f8e-ebc2-43a5-b8b5-72400aa31111-maas-api-tls\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:26:46.974376 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:46.974170    2548 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f79f8e-ebc2-43a5-b8b5-72400aa31111-openshift-service-ca\") on node \"ip-10-0-134-62.ec2.internal\" DevicePath \"\""
Sep 09 21:26:47.287789 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.287725    2548 generic.go:356] "Generic (PLEG): container finished" podID="65f79f8e-ebc2-43a5-b8b5-72400aa31111" containerID="18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15" exitCode=0
Sep 09 21:26:47.287971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.287793    2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7"
Sep 09 21:26:47.287971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.287821    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7" event={"ID":"65f79f8e-ebc2-43a5-b8b5-72400aa31111","Type":"ContainerDied","Data":"18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15"}
Sep 09 21:26:47.287971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.287879    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68d778f44-p8md7" event={"ID":"65f79f8e-ebc2-43a5-b8b5-72400aa31111","Type":"ContainerDied","Data":"4cd5e72c3d07404faa2a506be86bb2f23339814e49f9a1a81cf52ed11e590a03"}
Sep 09 21:26:47.287971 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.287900    2548 scope.go:117] "RemoveContainer" containerID="18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15"
Sep 09 21:26:47.297640 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.297623    2548 scope.go:117] "RemoveContainer" containerID="18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15"
Sep 09 21:26:47.297934 ip-10-0-134-62 kubenswrapper[2548]: E0909 21:26:47.297915    2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15\": container with ID starting with 18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15 not found: ID does not exist" containerID="18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15"
Sep 09 21:26:47.297988 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.297944    2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15"} err="failed to get container status \"18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15\": rpc error: code = NotFound desc = could not find container \"18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15\": container with ID starting with 18478e3147d69d044c9241c1f8980f72464994ce5090c7b8c8546842c26cfa15 not found: ID does not exist"
Sep 09 21:26:47.321401 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.321375    2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-p8md7"]
Sep 09 21:26:47.329520 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.329496    2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-68d778f44-p8md7"]
Sep 09 21:26:47.968476 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:47.968442    2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f79f8e-ebc2-43a5-b8b5-72400aa31111" path="/var/lib/kubelet/pods/65f79f8e-ebc2-43a5-b8b5-72400aa31111/volumes"
Sep 09 21:26:59.631072 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:26:59.631034    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dd5d6df4f-pm9zg_8b5fbd73-9b5a-47f2-accc-c0c9c22c841c/authorino/0.log"
Sep 09 21:27:04.840032 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:04.839993    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-smxn7_4751359f-6f18-4dca-ab21-69d8583c0bc3/manager/0.log"
Sep 09 21:27:04.963602 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:04.963570    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cd2bt_e6679e73-069b-424d-9cce-972deda45aa6/manager/0.log"
Sep 09 21:27:05.088329 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:05.088295    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f44688b45-47c9d_05314fe6-a0c2-401d-b453-0126895805c7/manager/0.log"
Sep 09 21:27:05.199977 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:05.199952    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ssnpl_34334705-4ee6-4798-b603-cbd03350e9c9/server/0.log"
Sep 09 21:27:05.322289 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:05.322257    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dgmjp_d8cd73f5-2805-43a9-990c-32f7192d450d/manager/0.log"
Sep 09 21:27:05.558406 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:05.558330    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-dq7m8_c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3/copy-manifests/0.log"
Sep 09 21:27:05.565353 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:05.565327    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-dq7m8_c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3/manager/0.log"
Sep 09 21:27:06.022075 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:06.022035    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-584858f848-7kkvc_0653f804-00a7-4e88-85fb-be8f56ce4be8/maas-api/0.log"
Sep 09 21:27:07.121004 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:07.120961    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dd5d6df4f-pm9zg_8b5fbd73-9b5a-47f2-accc-c0c9c22c841c/authorino/0.log"
Sep 09 21:27:07.235574 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:07.235543    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5mrdt_4fa60691-9cae-423a-a153-bed4e83bd0d4/manager/0.log"
Sep 09 21:27:07.342294 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:07.342265    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sblxn_ffa34442-1587-4aa9-ac6c-1190b0a3e7fc/manager/0.log"
Sep 09 21:27:07.579958 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:07.579908    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-vdzvc_f41ba5bb-2e87-474d-a7f6-a1057a2d881a/manager/0.log"
Sep 09 21:27:07.815650 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:07.815613    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-26vxx_8905ab88-e57b-45a3-9a35-e26b9e26b53a/manager/0.log"
Sep 09 21:27:08.154989 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:08.154959    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk_5917a147-4b51-484b-9b23-59733727a52e/istio-proxy/0.log"
Sep 09 21:27:08.485289 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:08.485261    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcf9fffd5-g8tlr_497cefe0-cfdd-46c6-9585-018befe67067/kube-auth-proxy/0.log"
Sep 09 21:27:08.610580 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:08.610552    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/1.log"
Sep 09 21:27:08.733081 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:08.733048    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-pvw2q_d4f3dc36-6f84-47e2-bfd9-a60c2350fea4/payload-pre-processing/0.log"
Sep 09 21:27:08.972273 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:08.972243    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6564b7c66d-qpftd_8100cb47-78a6-4eb4-9bfb-ccdc3025876f/router/0.log"
Sep 09 21:27:09.312412 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:09.312339    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kf7tl_065cb0cf-c562-4e61-ba16-33a90b50e89e/main/0.log"
Sep 09 21:27:09.427062 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:09.427031    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-khlwj_8bbe6b08-eef5-43f6-a921-d71ed0aa1b55/main/0.log"
Sep 09 21:27:09.970664 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:09.970635    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4kkjt_26f3d97f-7318-407b-adbf-83b5b9eb4598/main/0.log"
Sep 09 21:27:18.436267 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:18.436234    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-78qgz_813abdcc-2cfe-4ba7-95d4-42fce7903b66/global-pull-secret-syncer/0.log"
Sep 09 21:27:18.565571 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:18.565537    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qvj9k_99c43972-e7f2-4535-ae3a-205b5fded1e9/konnectivity-agent/0.log"
Sep 09 21:27:18.625513 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:18.625486    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-62.ec2.internal_6d6cefe1924478617631a1883cb3324d/haproxy/0.log"
Sep 09 21:27:23.392183 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:23.392131    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-dd5d6df4f-pm9zg_8b5fbd73-9b5a-47f2-accc-c0c9c22c841c/authorino/0.log"
Sep 09 21:27:23.425320 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:23.425291    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5mrdt_4fa60691-9cae-423a-a153-bed4e83bd0d4/manager/0.log"
Sep 09 21:27:23.450512 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:23.450465    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sblxn_ffa34442-1587-4aa9-ac6c-1190b0a3e7fc/manager/0.log"
Sep 09 21:27:23.535833 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:23.535804    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-vdzvc_f41ba5bb-2e87-474d-a7f6-a1057a2d881a/manager/0.log"
Sep 09 21:27:23.652653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:23.652585    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-26vxx_8905ab88-e57b-45a3-9a35-e26b9e26b53a/manager/0.log"
Sep 09 21:27:25.240850 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.240818    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-49fbc_1d6d21b7-47d9-423f-9127-f52cec9a1903/cluster-monitoring-operator/0.log"
Sep 09 21:27:25.661388 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.661309    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txtd2_635c8682-064a-424e-b68f-e7afac927bec/node-exporter/0.log"
Sep 09 21:27:25.683782 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.683760    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txtd2_635c8682-064a-424e-b68f-e7afac927bec/kube-rbac-proxy/0.log"
Sep 09 21:27:25.710016 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.709991    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txtd2_635c8682-064a-424e-b68f-e7afac927bec/init-textfile/0.log"
Sep 09 21:27:25.840829 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.840797    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/prometheus/0.log"
Sep 09 21:27:25.862494 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.862466    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/config-reloader/0.log"
Sep 09 21:27:25.885322 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.885301    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/thanos-sidecar/0.log"
Sep 09 21:27:25.910494 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.910474    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/kube-rbac-proxy-web/0.log"
Sep 09 21:27:25.943706 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:25.943685    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/kube-rbac-proxy/0.log"
Sep 09 21:27:26.006987 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:26.006970    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/kube-rbac-proxy-thanos/0.log"
Sep 09 21:27:26.063815 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:26.063795    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8bcd4a3-3b74-44c0-8d92-6e00fc010637/init-config-reloader/0.log"
Sep 09 21:27:28.465382 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.465352    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/1.log"
Sep 09 21:27:28.472984 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.472953    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fb6cz_1c71f27f-50a9-42a4-b6dd-21c58a14e5a3/console-operator/2.log"
Sep 09 21:27:28.660966 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.660938    2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"]
Sep 09 21:27:28.661301 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.661289    2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f79f8e-ebc2-43a5-b8b5-72400aa31111" containerName="maas-api"
Sep 09 21:27:28.661346 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.661303    2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f79f8e-ebc2-43a5-b8b5-72400aa31111" containerName="maas-api"
Sep 09 21:27:28.661387 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.661375    2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f79f8e-ebc2-43a5-b8b5-72400aa31111" containerName="maas-api"
Sep 09 21:27:28.664795 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.664774    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.667337 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.667308    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7p779\"/\"kube-root-ca.crt\""
Sep 09 21:27:28.667454 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.667321    2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7p779\"/\"openshift-service-ca.crt\""
Sep 09 21:27:28.668147 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.668131    2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7p779\"/\"default-dockercfg-fxqvc\""
Sep 09 21:27:28.677200 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.677180    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"]
Sep 09 21:27:28.830545 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.830458    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-podres\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.830545 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.830511    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-sys\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.830723 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.830582    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-lib-modules\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.830723 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.830627    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-proc\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.830723 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.830669    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbm9\" (UniqueName: \"kubernetes.io/projected/34d05323-82ec-4959-aeb1-ae13316cefb4-kube-api-access-2rbm9\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.931682 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931650    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbm9\" (UniqueName: \"kubernetes.io/projected/34d05323-82ec-4959-aeb1-ae13316cefb4-kube-api-access-2rbm9\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.931885 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931704    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-podres\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.931885 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931767    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-sys\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.931885 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931797    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-lib-modules\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.931885 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931840    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-proc\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.932095 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931899    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-sys\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.932095 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931936    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-proc\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.932095 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931936    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-podres\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.932095 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.931964    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34d05323-82ec-4959-aeb1-ae13316cefb4-lib-modules\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.941911 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.941886    2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbm9\" (UniqueName: \"kubernetes.io/projected/34d05323-82ec-4959-aeb1-ae13316cefb4-kube-api-access-2rbm9\") pod \"perf-node-gather-daemonset-5c8m6\" (UID: \"34d05323-82ec-4959-aeb1-ae13316cefb4\") " pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:28.974313 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:28.974287    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:29.243228 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:29.243201    2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"]
Sep 09 21:27:29.245100 ip-10-0-134-62 kubenswrapper[2548]: W0909 21:27:29.245073    2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod34d05323_82ec_4959_aeb1_ae13316cefb4.slice/crio-9016be6c5f3aaed5aa4e1ecca619097f5cf2063403c303e62b002b945ea1a328 WatchSource:0}: Error finding container 9016be6c5f3aaed5aa4e1ecca619097f5cf2063403c303e62b002b945ea1a328: Status 404 returned error can't find the container with id 9016be6c5f3aaed5aa4e1ecca619097f5cf2063403c303e62b002b945ea1a328
Sep 09 21:27:29.500106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:29.500034    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6" event={"ID":"34d05323-82ec-4959-aeb1-ae13316cefb4","Type":"ContainerStarted","Data":"65a668e089dfad5ef0f461fda14695b0f025b376ba29e89db6db964886573bb1"}
Sep 09 21:27:29.500106 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:29.500082    2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6" event={"ID":"34d05323-82ec-4959-aeb1-ae13316cefb4","Type":"ContainerStarted","Data":"9016be6c5f3aaed5aa4e1ecca619097f5cf2063403c303e62b002b945ea1a328"}
Sep 09 21:27:29.500450 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:29.500182    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:29.518947 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:29.518904    2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6" podStartSLOduration=1.518889967 podStartE2EDuration="1.518889967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:27:29.517283235 +0000 UTC m=+2338.020113270" watchObservedRunningTime="2026-09-09 21:27:29.518889967 +0000 UTC m=+2338.021719968"
Sep 09 21:27:31.209181 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:31.209147    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wl49r_ce390400-86f1-4cf0-8155-8313e802c2da/dns/0.log"
Sep 09 21:27:31.231104 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:31.231075    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wl49r_ce390400-86f1-4cf0-8155-8313e802c2da/kube-rbac-proxy/0.log"
Sep 09 21:27:31.254438 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:31.254418    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dkcxd_9dc7b235-2a09-46a5-9165-39e41d98d840/dns-node-resolver/0.log"
Sep 09 21:27:31.887653 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:31.887616    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-749b8b8858-dc7d7_db52dd3a-2f1a-4a0a-bfe4-1cb544b4c53a/registry/0.log"
Sep 09 21:27:31.934446 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:31.934415    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nsh2j_6619b496-637a-4621-a857-75c2fc6c97be/node-ca/0.log"
Sep 09 21:27:32.931373 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:32.931340    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk_5917a147-4b51-484b-9b23-59733727a52e/istio-proxy/0.log"
Sep 09 21:27:33.208851 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:33.208817    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcf9fffd5-g8tlr_497cefe0-cfdd-46c6-9585-018befe67067/kube-auth-proxy/0.log"
Sep 09 21:27:33.239253 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:33.239229    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/0.log"
Sep 09 21:27:33.254663 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:33.254642    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/1.log"
Sep 09 21:27:33.301444 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:33.301420    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-pvw2q_d4f3dc36-6f84-47e2-bfd9-a60c2350fea4/payload-pre-processing/0.log"
Sep 09 21:27:33.375763 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:33.375744    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6564b7c66d-qpftd_8100cb47-78a6-4eb4-9bfb-ccdc3025876f/router/0.log"
Sep 09 21:27:34.294489 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:34.294453    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xcvt7_086ea7e4-5ed3-41cc-9842-287c26d02fc7/serve-healthcheck-canary/0.log"
Sep 09 21:27:34.974710 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:34.974676    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkrpg_73d72e45-d692-4885-8d99-6fdb2f027312/kube-rbac-proxy/0.log"
Sep 09 21:27:34.998907 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:34.998879    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkrpg_73d72e45-d692-4885-8d99-6fdb2f027312/exporter/0.log"
Sep 09 21:27:35.022563 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:35.022542    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkrpg_73d72e45-d692-4885-8d99-6fdb2f027312/extractor/0.log"
Sep 09 21:27:35.515321 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:35.515293    2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7p779/perf-node-gather-daemonset-5c8m6"
Sep 09 21:27:37.312585 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:37.312556    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-smxn7_4751359f-6f18-4dca-ab21-69d8583c0bc3/manager/0.log"
Sep 09 21:27:37.619943 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:37.619862    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cd2bt_e6679e73-069b-424d-9cce-972deda45aa6/manager/0.log"
Sep 09 21:27:37.694685 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:37.694649    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f44688b45-47c9d_05314fe6-a0c2-401d-b453-0126895805c7/manager/0.log"
Sep 09 21:27:37.718640 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:37.718613    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ssnpl_34334705-4ee6-4798-b603-cbd03350e9c9/server/0.log"
Sep 09 21:27:37.856655 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:37.856627    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dgmjp_d8cd73f5-2805-43a9-990c-32f7192d450d/manager/0.log"
Sep 09 21:27:38.024629 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:38.024600    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-dq7m8_c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3/manager/0.log"
Sep 09 21:27:38.044440 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:38.044419    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d75fb655b-dq7m8_c2ac2e0a-c621-49cc-a57b-8c07ec7d29f3/copy-manifests/0.log"
Sep 09 21:27:39.338306 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:39.338277    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk_5917a147-4b51-484b-9b23-59733727a52e/istio-proxy/0.log"
Sep 09 21:27:39.580262 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:39.580234    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcf9fffd5-g8tlr_497cefe0-cfdd-46c6-9585-018befe67067/kube-auth-proxy/0.log"
Sep 09 21:27:39.603972 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:39.603886    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/0.log"
Sep 09 21:27:39.618652 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:39.618631    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/1.log"
Sep 09 21:27:39.657440 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:39.657406    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-pvw2q_d4f3dc36-6f84-47e2-bfd9-a60c2350fea4/payload-pre-processing/0.log"
Sep 09 21:27:39.723028 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:39.722987    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6564b7c66d-qpftd_8100cb47-78a6-4eb4-9bfb-ccdc3025876f/router/0.log"
Sep 09 21:27:41.209409 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:41.209375    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4rdwk_5917a147-4b51-484b-9b23-59733727a52e/istio-proxy/0.log"
Sep 09 21:27:41.457134 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:41.457103    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bcf9fffd5-g8tlr_497cefe0-cfdd-46c6-9585-018befe67067/kube-auth-proxy/0.log"
Sep 09 21:27:41.480690 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:41.480591    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/0.log"
Sep 09 21:27:41.495285 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:41.495261    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nrds4_ffd75c7e-ad26-454b-a660-e8e41929a0d5/istio-proxy/1.log"
Sep 09 21:27:41.532082 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:41.532057    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-pvw2q_d4f3dc36-6f84-47e2-bfd9-a60c2350fea4/payload-pre-processing/0.log"
Sep 09 21:27:41.599218 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:41.599191    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6564b7c66d-qpftd_8100cb47-78a6-4eb4-9bfb-ccdc3025876f/router/0.log"
Sep 09 21:27:44.887511 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:44.887412    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-tvf27_67f23aa7-d4b5-4d14-89a8-b1116f8c47b0/migrator/0.log"
Sep 09 21:27:44.907288 ip-10-0-134-62 kubenswrapper[2548]: I0909 21:27:44.907252    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-tvf27_67f23aa7-d4b5-4d14-89a8-b1116f8c47b0/graceful-termination/0.log"